[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 02:01:28.536628 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:46673,127.19.220.189:43193,127.19.220.188:34787
I20250626 02:01:28.537699 20338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:01:28.538164 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:28.544471 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.544600 20344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:28.544711 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:28.544471 20343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:28.545293 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:28.545409 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:28.545464 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903288545462 us; error 0 us; skew 500 ppm
I20250626 02:01:28.547309 20338 webserver.cc:469] Webserver started at http://127.19.220.190:44427/ using document root <none> and password file <none>
I20250626 02:01:28.547838 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:28.547894 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:28.548171 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:28.549819 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "c88547d378684ba5b16f065a90b49271"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.553282 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 02:01:28.555469 20351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.556501 20338 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250626 02:01:28.556593 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "c88547d378684ba5b16f065a90b49271"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.556684 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:28.581228 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:28.581867 20338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:01:28.581998 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:28.589052 20403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:46673 every 8 connection(s)
I20250626 02:01:28.589066 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:46673
I20250626 02:01:28.590256 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:28.591892 20409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.592015 20407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.591898 20406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:28.592187 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:28.592399 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:28.592454 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:28.592475 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903288592475 us; error 0 us; skew 500 ppm
I20250626 02:01:28.592550 20404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:28.593523 20338 webserver.cc:469] Webserver started at http://127.19.220.189:39179/ using document root <none> and password file <none>
I20250626 02:01:28.593688 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:28.593740 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:28.593809 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:28.594246 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root/instance:
uuid: "47b3137488f54bec980e9913c3bf700e"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.596145 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:28.597183 20414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.597440 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:28.597532 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root
uuid: "47b3137488f54bec980e9913c3bf700e"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.597594 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:28.597673 20404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:28.608462 20404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
W20250626 02:01:28.609148 20354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.220.189:43193: connect: Connection refused (error 111)
W20250626 02:01:28.609923 20404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.220.189:43193: Network error: Client connection negotiation failed: client connection to 127.19.220.189:43193: connect: Connection refused (error 111)
I20250626 02:01:28.611042 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:28.611387 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:28.616168 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.189:43193
I20250626 02:01:28.616261 20469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.189:43193 every 8 connection(s)
I20250626 02:01:28.616878 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:01:28.617044 20470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:28.618441 20470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
W20250626 02:01:28.618508 20472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.619594 20473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.619760 20475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:28.620121 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:28.620342 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:28.620388 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:28.620407 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903288620407 us; error 0 us; skew 500 ppm
I20250626 02:01:28.621335 20338 webserver.cc:469] Webserver started at http://127.19.220.188:46797/ using document root <none> and password file <none>
I20250626 02:01:28.621471 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:28.621522 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:28.621629 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:28.622097 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root/instance:
uuid: "bdb879bbe707436cb3b7325cb50b8818"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.622988 20470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:28.623900 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:28.624903 20482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.625144 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:28.625197 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root
uuid: "bdb879bbe707436cb3b7325cb50b8818"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.625268 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:28.627545 20470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } has no permanent_uuid. Determining permanent_uuid...
W20250626 02:01:28.628633 20470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.220.188:34787: Network error: Client connection negotiation failed: client connection to 127.19.220.188:34787: connect: Connection refused (error 111)
I20250626 02:01:28.631492 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:28.631817 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:28.636269 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.188:34787
I20250626 02:01:28.636312 20535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.188:34787 every 8 connection(s)
I20250626 02:01:28.636669 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:01:28.637072 20536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:28.638038 20536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:28.641748 20536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:28.645155 20536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:28.650534 20536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: Bootstrap starting.
I20250626 02:01:28.653232 20536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:28.654047 20536 log.cc:826] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: Log is configured to *not* fsync() on all Append() calls
I20250626 02:01:28.655500 20536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: No bootstrap required, opened a new log
I20250626 02:01:28.657796 20536 raft_consensus.cc:357] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.657943 20536 raft_consensus.cc:383] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:28.658015 20536 raft_consensus.cc:738] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb879bbe707436cb3b7325cb50b8818, State: Initialized, Role: FOLLOWER
I20250626 02:01:28.658389 20536 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.658790 20541 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.658890 20541 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:28.659087 20404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } attempt: 1
I20250626 02:01:28.659425 20536 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:01:28.663208 20404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } has no permanent_uuid. Determining permanent_uuid...
W20250626 02:01:28.664506 20552 catalog_manager.cc:1560] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 02:01:28.664574 20552 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 02:01:28.667976 20404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271: Bootstrap starting.
I20250626 02:01:28.668869 20404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:28.669870 20404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271: No bootstrap required, opened a new log
I20250626 02:01:28.670243 20404 raft_consensus.cc:357] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.670321 20404 raft_consensus.cc:383] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:28.670363 20404 raft_consensus.cc:738] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88547d378684ba5b16f065a90b49271, State: Initialized, Role: FOLLOWER
I20250626 02:01:28.670498 20404 consensus_queue.cc:260] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.670773 20555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.670878 20555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:28.670985 20404 sys_catalog.cc:564] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:01:28.672751 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250626 02:01:28.673215 20566 catalog_manager.cc:1560] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 02:01:28.673272 20566 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 02:01:28.680779 20470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } attempt: 1
I20250626 02:01:28.685274 20470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e: Bootstrap starting.
I20250626 02:01:28.686141 20470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:28.687398 20470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e: No bootstrap required, opened a new log
I20250626 02:01:28.687789 20470 raft_consensus.cc:357] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.687873 20470 raft_consensus.cc:383] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:28.687908 20470 raft_consensus.cc:738] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b3137488f54bec980e9913c3bf700e, State: Initialized, Role: FOLLOWER
I20250626 02:01:28.688040 20470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.688396 20568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.688488 20470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:01:28.688496 20568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:28.690301 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250626 02:01:28.690811 20579 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 02:01:28.690873 20579 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 02:01:28.690924 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:28.693358 20581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.693411 20580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.693518 20583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:28.693569 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:28.693744 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:28.693787 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:28.693816 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903288693815 us; error 0 us; skew 500 ppm
I20250626 02:01:28.694809 20338 webserver.cc:469] Webserver started at http://127.19.220.129:33551/ using document root <none> and password file <none>
I20250626 02:01:28.694979 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:28.695032 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:28.695102 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:28.695549 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "c137637ba06e446ab01acadd203288f2"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.697238 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:01:28.698268 20588 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.698529 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:28.698597 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "c137637ba06e446ab01acadd203288f2"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.698658 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:28.716601 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:28.717087 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:28.717396 20568 raft_consensus.cc:491] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:28.717576 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:28.717594 20568 raft_consensus.cc:513] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.718511 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:28.718570 20568 leader_election.cc:290] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c88547d378684ba5b16f065a90b49271 (127.19.220.190:46673), bdb879bbe707436cb3b7325cb50b8818 (127.19.220.188:34787)
I20250626 02:01:28.718619 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.718674 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:28.718698 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.719094 20378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47b3137488f54bec980e9913c3bf700e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88547d378684ba5b16f065a90b49271" is_pre_election: true
I20250626 02:01:28.719117 20510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47b3137488f54bec980e9913c3bf700e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb879bbe707436cb3b7325cb50b8818" is_pre_election: true
I20250626 02:01:28.719318 20378 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b3137488f54bec980e9913c3bf700e in term 0.
I20250626 02:01:28.719318 20510 raft_consensus.cc:2466] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b3137488f54bec980e9913c3bf700e in term 0.
I20250626 02:01:28.719740 20418 leader_election.cc:304] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b3137488f54bec980e9913c3bf700e, bdb879bbe707436cb3b7325cb50b8818; no voters:
I20250626 02:01:28.719923 20568 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:28.719972 20568 raft_consensus.cc:491] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:28.720009 20568 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:28.721479 20568 raft_consensus.cc:513] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.721861 20568 leader_election.cc:290] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [CANDIDATE]: Term 1 election: Requested vote from peers c88547d378684ba5b16f065a90b49271 (127.19.220.190:46673), bdb879bbe707436cb3b7325cb50b8818 (127.19.220.188:34787)
I20250626 02:01:28.722271 20510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47b3137488f54bec980e9913c3bf700e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb879bbe707436cb3b7325cb50b8818"
I20250626 02:01:28.722365 20510 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:28.722642 20378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47b3137488f54bec980e9913c3bf700e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88547d378684ba5b16f065a90b49271"
I20250626 02:01:28.722728 20378 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:28.723337 20510 raft_consensus.cc:2466] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b3137488f54bec980e9913c3bf700e in term 1.
I20250626 02:01:28.723667 20378 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b3137488f54bec980e9913c3bf700e in term 1.
I20250626 02:01:28.723701 20418 leader_election.cc:304] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b3137488f54bec980e9913c3bf700e, bdb879bbe707436cb3b7325cb50b8818; no voters:
I20250626 02:01:28.724083 20568 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:28.724288 20568 raft_consensus.cc:695] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 1 LEADER]: Becoming Leader. State: Replica: 47b3137488f54bec980e9913c3bf700e, State: Running, Role: LEADER
I20250626 02:01:28.724577 20568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } }
I20250626 02:01:28.724864 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:43965
I20250626 02:01:28.724907 20652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:43965 every 8 connection(s)
I20250626 02:01:28.725948 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:01:28.726506 20647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b3137488f54bec980e9913c3bf700e. Latest consensus state: current_term: 1 leader_uuid: "47b3137488f54bec980e9913c3bf700e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.726596 20647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:28.726956 20659 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:01:28.728046 20659 catalog_manager.cc:1486] Initializing Kudu cluster ID...
W20250626 02:01:28.730255 20660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:28.731652 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:28.732295 20655 heartbeater.cc:344] Connected to a master server at 127.19.220.189:43193
I20250626 02:01:28.732442 20655 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.732998 20655 heartbeater.cc:507] Master 127.19.220.189:43193 requested a full tablet report, sending...
I20250626 02:01:28.734308 20434 ts_manager.cc:194] Registered new tserver with Master: c137637ba06e446ab01acadd203288f2 (127.19.220.129:43965)
I20250626 02:01:28.734486 20510 raft_consensus.cc:1273] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 1 FOLLOWER]: Refusing update from remote peer 47b3137488f54bec980e9913c3bf700e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:01:28.734901 20568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:28.735694 20378 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 1 FOLLOWER]: Refusing update from remote peer 47b3137488f54bec980e9913c3bf700e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 02:01:28.736249 20663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.737282 20661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:28.737385 20541 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b3137488f54bec980e9913c3bf700e. Latest consensus state: current_term: 1 leader_uuid: "47b3137488f54bec980e9913c3bf700e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.737504 20541 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:28.737521 20654 heartbeater.cc:344] Connected to a master server at 127.19.220.190:46673
I20250626 02:01:28.737553 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:28.737586 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:28.737612 20653 heartbeater.cc:344] Connected to a master server at 127.19.220.188:34787
I20250626 02:01:28.737625 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903288737625 us; error 0 us; skew 500 ppm
I20250626 02:01:28.737643 20568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:28.738711 20338 webserver.cc:469] Webserver started at http://127.19.220.130:44031/ using document root <none> and password file <none>
I20250626 02:01:28.738796 20555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47b3137488f54bec980e9913c3bf700e. Latest consensus state: current_term: 1 leader_uuid: "47b3137488f54bec980e9913c3bf700e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.738888 20555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:28.738909 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:28.738960 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:28.737653 20653 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.739053 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:28.739285 20568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47b3137488f54bec980e9913c3bf700e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.739385 20568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:28.739271 20653 heartbeater.cc:507] Master 127.19.220.188:34787 requested a full tablet report, sending...
I20250626 02:01:28.739568 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "fb39bf1853aa40aba75b577c4f5e41c2"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.740025 20541 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47b3137488f54bec980e9913c3bf700e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.740136 20541 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:28.737591 20654 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.740392 20654 heartbeater.cc:507] Master 127.19.220.190:46673 requested a full tablet report, sending...
I20250626 02:01:28.740104 20500 ts_manager.cc:194] Registered new tserver with Master: c137637ba06e446ab01acadd203288f2 (127.19.220.129:43965)
I20250626 02:01:28.740998 20368 ts_manager.cc:194] Registered new tserver with Master: c137637ba06e446ab01acadd203288f2 (127.19.220.129:43965)
I20250626 02:01:28.740993 20667 mvcc.cc:204] Tried to move back new op lower bound from 7171699870649987072 to 7171699870618673152. Current Snapshot: MvccSnapshot[applied={T|T < 7171699870649987072}]
I20250626 02:01:28.742081 20568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47b3137488f54bec980e9913c3bf700e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.742165 20568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:28.742897 20555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47b3137488f54bec980e9913c3bf700e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88547d378684ba5b16f065a90b49271" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 46673 } } peers { permanent_uuid: "47b3137488f54bec980e9913c3bf700e" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 43193 } } peers { permanent_uuid: "bdb879bbe707436cb3b7325cb50b8818" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 34787 } } }
I20250626 02:01:28.742988 20555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:28.743496 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20250626 02:01:28.744644 20659 catalog_manager.cc:1349] Generated new cluster ID: 46f49db6d93043df8673698d562fb5c0
I20250626 02:01:28.744695 20659 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:01:28.745090 20678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.745301 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:28.745357 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "fb39bf1853aa40aba75b577c4f5e41c2"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.745419 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:28.751466 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:28.751885 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:28.752189 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:28.752835 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:28.753656 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:28.753724 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:28.753751 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.755355 20659 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:01:28.756505 20659 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:01:28.759004 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:42633
I20250626 02:01:28.760169 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:01:28.762136 20742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:42633 every 8 connection(s)
I20250626 02:01:28.767062 20745 heartbeater.cc:344] Connected to a master server at 127.19.220.189:43193
I20250626 02:01:28.767150 20745 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.767347 20745 heartbeater.cc:507] Master 127.19.220.189:43193 requested a full tablet report, sending...
I20250626 02:01:28.767931 20434 ts_manager.cc:194] Registered new tserver with Master: fb39bf1853aa40aba75b577c4f5e41c2 (127.19.220.130:42633)
I20250626 02:01:28.768916 20743 heartbeater.cc:344] Connected to a master server at 127.19.220.188:34787
I20250626 02:01:28.769011 20743 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.769222 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:28.769246 20743 heartbeater.cc:507] Master 127.19.220.188:34787 requested a full tablet report, sending...
W20250626 02:01:28.768945 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.769531 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:28.769671 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:28.769856 20500 ts_manager.cc:194] Registered new tserver with Master: fb39bf1853aa40aba75b577c4f5e41c2 (127.19.220.130:42633)
I20250626 02:01:28.769999 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:28.770041 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:28.770069 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903288770069 us; error 0 us; skew 500 ppm
I20250626 02:01:28.771420 20338 webserver.cc:469] Webserver started at http://127.19.220.131:37207/ using document root <none> and password file <none>
I20250626 02:01:28.771577 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:28.771627 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:28.771694 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:28.772115 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "a4afc50bf1b5481e975ce1360496b99b"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.773808 20659 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e: Generated new TSK 0
I20250626 02:01:28.774549 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250626 02:01:28.774688 20659 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:01:28.775416 20744 heartbeater.cc:344] Connected to a master server at 127.19.220.190:46673
I20250626 02:01:28.775506 20744 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.775692 20758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.775720 20744 heartbeater.cc:507] Master 127.19.220.190:46673 requested a full tablet report, sending...
I20250626 02:01:28.775925 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:28.775988 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "a4afc50bf1b5481e975ce1360496b99b"
format_stamp: "Formatted at 2025-06-26 02:01:28 on dist-test-slave-h18f"
I20250626 02:01:28.776059 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:28.776289 20368 ts_manager.cc:194] Registered new tserver with Master: fb39bf1853aa40aba75b577c4f5e41c2 (127.19.220.130:42633)
I20250626 02:01:28.789486 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:28.789862 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:28.790235 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:28.790706 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:28.790745 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.790797 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:28.790831 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:28.795804 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:36523
I20250626 02:01:28.795890 20821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:36523 every 8 connection(s)
I20250626 02:01:28.801501 20822 heartbeater.cc:344] Connected to a master server at 127.19.220.188:34787
I20250626 02:01:28.801607 20822 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.801792 20823 heartbeater.cc:344] Connected to a master server at 127.19.220.190:46673
I20250626 02:01:28.801829 20822 heartbeater.cc:507] Master 127.19.220.188:34787 requested a full tablet report, sending...
I20250626 02:01:28.801885 20823 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.802090 20823 heartbeater.cc:507] Master 127.19.220.190:46673 requested a full tablet report, sending...
I20250626 02:01:28.802440 20500 ts_manager.cc:194] Registered new tserver with Master: a4afc50bf1b5481e975ce1360496b99b (127.19.220.131:36523)
I20250626 02:01:28.802982 20368 ts_manager.cc:194] Registered new tserver with Master: a4afc50bf1b5481e975ce1360496b99b (127.19.220.131:36523)
I20250626 02:01:28.807691 20824 heartbeater.cc:344] Connected to a master server at 127.19.220.189:43193
I20250626 02:01:28.807772 20824 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:28.807960 20824 heartbeater.cc:507] Master 127.19.220.189:43193 requested a full tablet report, sending...
I20250626 02:01:28.808498 20434 ts_manager.cc:194] Registered new tserver with Master: a4afc50bf1b5481e975ce1360496b99b (127.19.220.131:36523)
I20250626 02:01:28.808669 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.0122429s
I20250626 02:01:28.809572 20434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53438
I20250626 02:01:29.665795 20552 catalog_manager.cc:1261] Loaded cluster ID: 46f49db6d93043df8673698d562fb5c0
I20250626 02:01:29.665886 20552 catalog_manager.cc:1554] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: loading cluster ID for follower catalog manager: success
I20250626 02:01:29.667138 20552 catalog_manager.cc:1576] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: acquiring CA information for follower catalog manager: success
I20250626 02:01:29.667747 20552 catalog_manager.cc:1604] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 02:01:29.673857 20566 catalog_manager.cc:1261] Loaded cluster ID: 46f49db6d93043df8673698d562fb5c0
I20250626 02:01:29.673928 20566 catalog_manager.cc:1554] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271: loading cluster ID for follower catalog manager: success
I20250626 02:01:29.674979 20566 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271: acquiring CA information for follower catalog manager: success
I20250626 02:01:29.675446 20566 catalog_manager.cc:1604] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 02:01:29.736930 20434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53416
I20250626 02:01:29.769930 20434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53430
I20250626 02:01:29.810689 20824 heartbeater.cc:499] Master 127.19.220.189:43193 was elected leader, sending a full tablet report...
I20250626 02:01:30.690505 20578 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:30.737962 20655 heartbeater.cc:499] Master 127.19.220.189:43193 was elected leader, sending a full tablet report...
I20250626 02:01:30.770877 20745 heartbeater.cc:499] Master 127.19.220.189:43193 was elected leader, sending a full tablet report...
I20250626 02:01:30.862890 20338 test_util.cc:276] Using random seed: -256893660
I20250626 02:01:30.875131 20434 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53452:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 02:01:30.892287 20705 tablet_service.cc:1468] Processing CreateTablet for tablet f37c89cf1a3e44e5af7766fab4e24c1f (DEFAULT_TABLE table=test-workload [id=5647283561d04056b8cf02b62db63d66]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:01:30.892494 20786 tablet_service.cc:1468] Processing CreateTablet for tablet 69dac9f174c8451ea50d42b04d834a8c (DEFAULT_TABLE table=test-workload [id=5647283561d04056b8cf02b62db63d66]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:01:30.892750 20786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69dac9f174c8451ea50d42b04d834a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:30.892750 20705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f37c89cf1a3e44e5af7766fab4e24c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:30.895615 20608 tablet_service.cc:1468] Processing CreateTablet for tablet 0be73bdbe0ec46519d4cd5d3b0d77d87 (DEFAULT_TABLE table=test-workload [id=5647283561d04056b8cf02b62db63d66]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:01:30.895679 20616 tablet_service.cc:1468] Processing CreateTablet for tablet ec917b7a0cbf427e8a524418a5b12f69 (DEFAULT_TABLE table=test-workload [id=5647283561d04056b8cf02b62db63d66]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:01:30.895746 20856 tablet_bootstrap.cc:492] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b: Bootstrap starting.
I20250626 02:01:30.895891 20857 tablet_bootstrap.cc:492] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2: Bootstrap starting.
I20250626 02:01:30.895886 20608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be73bdbe0ec46519d4cd5d3b0d77d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:30.896188 20616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec917b7a0cbf427e8a524418a5b12f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:30.896870 20856 tablet_bootstrap.cc:654] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:30.896872 20857 tablet_bootstrap.cc:654] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:30.897951 20857 tablet_bootstrap.cc:492] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2: No bootstrap required, opened a new log
I20250626 02:01:30.898046 20857 ts_tablet_manager.cc:1397] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:30.898389 20860 tablet_bootstrap.cc:492] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2: Bootstrap starting.
I20250626 02:01:30.898519 20857 raft_consensus.cc:357] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39bf1853aa40aba75b577c4f5e41c2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 42633 } }
I20250626 02:01:30.898624 20857 raft_consensus.cc:383] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:30.898669 20857 raft_consensus.cc:738] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb39bf1853aa40aba75b577c4f5e41c2, State: Initialized, Role: FOLLOWER
I20250626 02:01:30.898798 20857 consensus_queue.cc:260] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39bf1853aa40aba75b577c4f5e41c2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 42633 } }
I20250626 02:01:30.898905 20857 raft_consensus.cc:397] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:30.898954 20857 raft_consensus.cc:491] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:30.899006 20857 raft_consensus.cc:3058] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:30.899220 20860 tablet_bootstrap.cc:654] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:30.899709 20857 raft_consensus.cc:513] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39bf1853aa40aba75b577c4f5e41c2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 42633 } }
I20250626 02:01:30.899840 20857 leader_election.cc:304] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb39bf1853aa40aba75b577c4f5e41c2; no voters:
I20250626 02:01:30.900002 20857 leader_election.cc:290] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:30.900156 20862 raft_consensus.cc:2802] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:30.900283 20860 tablet_bootstrap.cc:492] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2: No bootstrap required, opened a new log
I20250626 02:01:30.900323 20857 ts_tablet_manager.cc:1428] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:30.900372 20860 ts_tablet_manager.cc:1397] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:01:30.900446 20862 raft_consensus.cc:695] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 1 LEADER]: Becoming Leader. State: Replica: fb39bf1853aa40aba75b577c4f5e41c2, State: Running, Role: LEADER
I20250626 02:01:30.901042 20856 tablet_bootstrap.cc:492] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b: No bootstrap required, opened a new log
I20250626 02:01:30.901031 20860 raft_consensus.cc:357] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } }
I20250626 02:01:30.901001 20862 consensus_queue.cc:237] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39bf1853aa40aba75b577c4f5e41c2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 42633 } }
I20250626 02:01:30.901163 20860 raft_consensus.cc:383] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:30.901219 20860 raft_consensus.cc:738] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c137637ba06e446ab01acadd203288f2, State: Initialized, Role: FOLLOWER
I20250626 02:01:30.901343 20860 consensus_queue.cc:260] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } }
I20250626 02:01:30.901425 20860 raft_consensus.cc:397] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:30.901460 20860 raft_consensus.cc:491] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:30.901512 20860 raft_consensus.cc:3058] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:30.902292 20860 raft_consensus.cc:513] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } }
I20250626 02:01:30.902415 20860 leader_election.cc:304] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c137637ba06e446ab01acadd203288f2; no voters:
I20250626 02:01:30.902606 20860 leader_election.cc:290] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:30.902714 20866 raft_consensus.cc:2802] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:30.902884 20866 raft_consensus.cc:695] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 1 LEADER]: Becoming Leader. State: Replica: c137637ba06e446ab01acadd203288f2, State: Running, Role: LEADER
I20250626 02:01:30.902961 20860 ts_tablet_manager.cc:1428] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:01:30.902993 20866 consensus_queue.cc:237] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } }
I20250626 02:01:30.903137 20860 tablet_bootstrap.cc:492] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2: Bootstrap starting.
I20250626 02:01:30.903955 20860 tablet_bootstrap.cc:654] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:30.904498 20434 catalog_manager.cc:5582] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb39bf1853aa40aba75b577c4f5e41c2 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "fb39bf1853aa40aba75b577c4f5e41c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb39bf1853aa40aba75b577c4f5e41c2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 42633 } health_report { overall_health: HEALTHY } } }
I20250626 02:01:30.905162 20856 ts_tablet_manager.cc:1397] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250626 02:01:30.905504 20856 raft_consensus.cc:357] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4afc50bf1b5481e975ce1360496b99b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36523 } }
I20250626 02:01:30.905562 20856 raft_consensus.cc:383] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:30.905587 20856 raft_consensus.cc:738] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4afc50bf1b5481e975ce1360496b99b, State: Initialized, Role: FOLLOWER
I20250626 02:01:30.905627 20433 catalog_manager.cc:5582] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c137637ba06e446ab01acadd203288f2 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "c137637ba06e446ab01acadd203288f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } health_report { overall_health: HEALTHY } } }
I20250626 02:01:30.905694 20856 consensus_queue.cc:260] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4afc50bf1b5481e975ce1360496b99b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36523 } }
I20250626 02:01:30.905778 20856 raft_consensus.cc:397] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:30.905812 20856 raft_consensus.cc:491] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:30.905858 20856 raft_consensus.cc:3058] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:30.906435 20856 raft_consensus.cc:513] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4afc50bf1b5481e975ce1360496b99b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36523 } }
I20250626 02:01:30.906546 20856 leader_election.cc:304] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4afc50bf1b5481e975ce1360496b99b; no voters:
I20250626 02:01:30.906731 20856 leader_election.cc:290] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:30.907030 20856 ts_tablet_manager.cc:1428] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:30.910266 20860 tablet_bootstrap.cc:492] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2: No bootstrap required, opened a new log
I20250626 02:01:30.910362 20860 ts_tablet_manager.cc:1397] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20250626 02:01:30.910784 20860 raft_consensus.cc:357] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } }
I20250626 02:01:30.910877 20860 raft_consensus.cc:383] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:30.910910 20860 raft_consensus.cc:738] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c137637ba06e446ab01acadd203288f2, State: Initialized, Role: FOLLOWER
I20250626 02:01:30.911044 20860 consensus_queue.cc:260] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } }
I20250626 02:01:30.911134 20860 raft_consensus.cc:397] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:30.911170 20860 raft_consensus.cc:491] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:30.911208 20860 raft_consensus.cc:3058] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:30.912075 20860 raft_consensus.cc:513] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } }
I20250626 02:01:30.912218 20860 leader_election.cc:304] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c137637ba06e446ab01acadd203288f2; no voters:
I20250626 02:01:30.912307 20860 leader_election.cc:290] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:30.912325 20874 raft_consensus.cc:2802] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:30.912456 20867 raft_consensus.cc:2802] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:30.912564 20860 ts_tablet_manager.cc:1428] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:30.912551 20867 raft_consensus.cc:695] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 1 LEADER]: Becoming Leader. State: Replica: c137637ba06e446ab01acadd203288f2, State: Running, Role: LEADER
I20250626 02:01:30.912511 20874 raft_consensus.cc:695] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 1 LEADER]: Becoming Leader. State: Replica: a4afc50bf1b5481e975ce1360496b99b, State: Running, Role: LEADER
I20250626 02:01:30.912820 20867 consensus_queue.cc:237] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } }
I20250626 02:01:30.912848 20874 consensus_queue.cc:237] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4afc50bf1b5481e975ce1360496b99b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36523 } }
I20250626 02:01:30.914919 20433 catalog_manager.cc:5582] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c137637ba06e446ab01acadd203288f2 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "c137637ba06e446ab01acadd203288f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c137637ba06e446ab01acadd203288f2" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43965 } health_report { overall_health: HEALTHY } } }
I20250626 02:01:30.916215 20434 catalog_manager.cc:5582] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b reported cstate change: term changed from 0 to 1, leader changed from <none> to a4afc50bf1b5481e975ce1360496b99b (127.19.220.131). New cstate: current_term: 1 leader_uuid: "a4afc50bf1b5481e975ce1360496b99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4afc50bf1b5481e975ce1360496b99b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36523 } health_report { overall_health: HEALTHY } } }
I20250626 02:01:30.921634 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:01:30.925920 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:30.926179 20338 tablet_replica.cc:331] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2: stopping tablet replica
I20250626 02:01:30.926433 20338 raft_consensus.cc:2241] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:30.926554 20338 pending_rounds.cc:70] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2: Trying to abort 1 pending ops.
I20250626 02:01:30.926609 20338 pending_rounds.cc:77] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171699879579361280 op_type: NO_OP noop_request { }
I20250626 02:01:30.926681 20338 raft_consensus.cc:2887] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:01:30.926744 20338 raft_consensus.cc:2270] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 02:01:30.948614 20877 consensus_queue.cc:1162] T ec917b7a0cbf427e8a524418a5b12f69 P c137637ba06e446ab01acadd203288f2 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c137637ba06e446ab01acadd203288f2" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250626 02:01:30.949697 20338 tablet_replica.cc:331] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2: stopping tablet replica
I20250626 02:01:30.949820 20338 raft_consensus.cc:2241] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:30.949936 20338 raft_consensus.cc:2270] T 0be73bdbe0ec46519d4cd5d3b0d77d87 P c137637ba06e446ab01acadd203288f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:30.963938 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:01:30.967686 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:01:30.971217 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:30.971457 20338 tablet_replica.cc:331] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2: stopping tablet replica
I20250626 02:01:30.971582 20338 raft_consensus.cc:2241] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:30.971704 20338 raft_consensus.cc:2270] T f37c89cf1a3e44e5af7766fab4e24c1f P fb39bf1853aa40aba75b577c4f5e41c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:30.984669 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:01:30.986481 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:01:30.989910 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:30.990180 20338 tablet_replica.cc:331] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b: stopping tablet replica
I20250626 02:01:30.990280 20338 raft_consensus.cc:2241] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:30.990389 20338 raft_consensus.cc:2270] T 69dac9f174c8451ea50d42b04d834a8c P a4afc50bf1b5481e975ce1360496b99b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:31.003376 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:01:31.005151 20338 master.cc:561] Master@127.19.220.190:46673 shutting down...
I20250626 02:01:31.008445 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:31.008599 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:31.008672 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P c88547d378684ba5b16f065a90b49271: stopping tablet replica
I20250626 02:01:31.021632 20338 master.cc:583] Master@127.19.220.190:46673 shutdown complete.
I20250626 02:01:31.024046 20338 master.cc:561] Master@127.19.220.189:43193 shutting down...
I20250626 02:01:31.027179 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:31.027511 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:31.027578 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P 47b3137488f54bec980e9913c3bf700e: stopping tablet replica
I20250626 02:01:31.040695 20338 master.cc:583] Master@127.19.220.189:43193 shutdown complete.
I20250626 02:01:31.043318 20338 master.cc:561] Master@127.19.220.188:34787 shutting down...
I20250626 02:01:31.046473 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:31.046618 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:31.046685 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P bdb879bbe707436cb3b7325cb50b8818: stopping tablet replica
I20250626 02:01:31.059386 20338 master.cc:583] Master@127.19.220.188:34787 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2530 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20250626 02:01:31.066732 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:34233
I20250626 02:01:31.067059 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:31.068576 20880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:31.068576 20881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:31.068706 20883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:31.068969 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:31.069098 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:31.069133 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:31.069154 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903291069154 us; error 0 us; skew 500 ppm
I20250626 02:01:31.070055 20338 webserver.cc:469] Webserver started at http://127.19.220.190:40927/ using document root <none> and password file <none>
I20250626 02:01:31.070200 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:31.070252 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:31.070330 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:31.070704 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "d1da6d3852a3414baa28859e57edf1fd"
format_stamp: "Formatted at 2025-06-26 02:01:31 on dist-test-slave-h18f"
I20250626 02:01:31.072119 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:31.073072 20888 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.073328 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:31.073405 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "d1da6d3852a3414baa28859e57edf1fd"
format_stamp: "Formatted at 2025-06-26 02:01:31 on dist-test-slave-h18f"
I20250626 02:01:31.073470 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:31.090708 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:31.091075 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:31.095098 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:34233
I20250626 02:01:31.098089 20940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:34233 every 8 connection(s)
I20250626 02:01:31.098619 20941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:31.100392 20941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd: Bootstrap starting.
I20250626 02:01:31.101121 20941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:31.102082 20941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd: No bootstrap required, opened a new log
I20250626 02:01:31.102430 20941 raft_consensus.cc:357] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da6d3852a3414baa28859e57edf1fd" member_type: VOTER }
I20250626 02:01:31.102510 20941 raft_consensus.cc:383] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:31.102532 20941 raft_consensus.cc:738] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da6d3852a3414baa28859e57edf1fd, State: Initialized, Role: FOLLOWER
I20250626 02:01:31.102636 20941 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da6d3852a3414baa28859e57edf1fd" member_type: VOTER }
I20250626 02:01:31.102721 20941 raft_consensus.cc:397] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:31.102754 20941 raft_consensus.cc:491] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:31.102790 20941 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:31.103412 20941 raft_consensus.cc:513] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da6d3852a3414baa28859e57edf1fd" member_type: VOTER }
I20250626 02:01:31.103518 20941 leader_election.cc:304] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1da6d3852a3414baa28859e57edf1fd; no voters:
I20250626 02:01:31.103654 20941 leader_election.cc:290] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:31.103775 20944 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:31.103955 20944 raft_consensus.cc:695] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 1 LEADER]: Becoming Leader. State: Replica: d1da6d3852a3414baa28859e57edf1fd, State: Running, Role: LEADER
I20250626 02:01:31.104081 20944 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da6d3852a3414baa28859e57edf1fd" member_type: VOTER }
I20250626 02:01:31.104085 20941 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:01:31.104568 20945 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1da6d3852a3414baa28859e57edf1fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da6d3852a3414baa28859e57edf1fd" member_type: VOTER } }
I20250626 02:01:31.104660 20945 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:31.104620 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1da6d3852a3414baa28859e57edf1fd. Latest consensus state: current_term: 1 leader_uuid: "d1da6d3852a3414baa28859e57edf1fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1da6d3852a3414baa28859e57edf1fd" member_type: VOTER } }
I20250626 02:01:31.104707 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:31.104887 20949 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:01:31.105718 20949 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:01:31.106055 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:01:31.107425 20949 catalog_manager.cc:1349] Generated new cluster ID: 79563db08d9a451884a84a66b0281959
I20250626 02:01:31.107481 20949 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:01:31.111932 20949 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:01:31.112465 20949 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:01:31.128499 20949 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd: Generated new TSK 0
I20250626 02:01:31.128674 20949 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:01:31.138417 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:31.140220 20963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:31.140287 20965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:31.140338 20962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:31.140484 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:31.140622 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:31.140658 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:31.140674 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903291140674 us; error 0 us; skew 500 ppm
I20250626 02:01:31.141587 20338 webserver.cc:469] Webserver started at http://127.19.220.129:45495/ using document root <none> and password file <none>
I20250626 02:01:31.141726 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:31.141772 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:31.141856 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:31.142282 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "2afdb4ad450b40fbba9bf06deaf50715"
format_stamp: "Formatted at 2025-06-26 02:01:31 on dist-test-slave-h18f"
I20250626 02:01:31.143661 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:31.144580 20970 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.144776 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:31.144840 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "2afdb4ad450b40fbba9bf06deaf50715"
format_stamp: "Formatted at 2025-06-26 02:01:31 on dist-test-slave-h18f"
I20250626 02:01:31.144915 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:31.149845 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:31.150199 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:31.150512 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:31.151044 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:31.151085 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.151129 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:31.151163 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.155481 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:45863
I20250626 02:01:31.155531 21033 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:45863 every 8 connection(s)
I20250626 02:01:31.156231 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:31.157732 21039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:31.157814 21038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:31.158056 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:31.158277 21041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:31.158483 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:31.158564 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:31.158586 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903291158586 us; error 0 us; skew 500 ppm
I20250626 02:01:31.159528 20338 webserver.cc:469] Webserver started at http://127.19.220.130:38009/ using document root <none> and password file <none>
I20250626 02:01:31.159663 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:31.159706 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:31.159772 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:31.160219 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "c089d4f3d49d4a24a66478348d576138"
format_stamp: "Formatted at 2025-06-26 02:01:31 on dist-test-slave-h18f"
I20250626 02:01:31.161584 21034 heartbeater.cc:344] Connected to a master server at 127.19.220.190:34233
I20250626 02:01:31.161665 21034 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:31.161892 21034 heartbeater.cc:507] Master 127.19.220.190:34233 requested a full tablet report, sending...
I20250626 02:01:31.161959 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250626 02:01:31.162462 20905 ts_manager.cc:194] Registered new tserver with Master: 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863)
I20250626 02:01:31.162930 21046 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.163225 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:31.163295 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "c089d4f3d49d4a24a66478348d576138"
format_stamp: "Formatted at 2025-06-26 02:01:31 on dist-test-slave-h18f"
I20250626 02:01:31.163303 20905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33452
I20250626 02:01:31.163379 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:31.172896 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:31.173220 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:31.173475 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:31.173873 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:31.173908 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.173955 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:31.173987 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.177858 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:46001
I20250626 02:01:31.177927 21109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:46001 every 8 connection(s)
I20250626 02:01:31.178589 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:31.180303 21113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:31.180301 21114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:31.180558 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:31.180783 21116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:31.180998 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:31.181056 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:31.181079 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903291181079 us; error 0 us; skew 500 ppm
I20250626 02:01:31.182293 20338 webserver.cc:469] Webserver started at http://127.19.220.131:38721/ using document root <none> and password file <none>
I20250626 02:01:31.182467 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:31.182519 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:31.182610 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:31.182828 21110 heartbeater.cc:344] Connected to a master server at 127.19.220.190:34233
I20250626 02:01:31.182919 21110 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:31.183053 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "93b029e769a34e469c6bc80ad083f28b"
format_stamp: "Formatted at 2025-06-26 02:01:31 on dist-test-slave-h18f"
I20250626 02:01:31.183096 21110 heartbeater.cc:507] Master 127.19.220.190:34233 requested a full tablet report, sending...
I20250626 02:01:31.183670 20905 ts_manager.cc:194] Registered new tserver with Master: c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001)
I20250626 02:01:31.184186 20905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33468
I20250626 02:01:31.184758 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:01:31.185649 21121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.185909 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:31.185969 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "93b029e769a34e469c6bc80ad083f28b"
format_stamp: "Formatted at 2025-06-26 02:01:31 on dist-test-slave-h18f"
I20250626 02:01:31.186040 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:31.190779 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:31.191103 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:31.191368 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:31.191756 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:31.191792 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.191825 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:31.191856 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:31.196396 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:45181
I20250626 02:01:31.196455 21184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:45181 every 8 connection(s)
I20250626 02:01:31.200520 21185 heartbeater.cc:344] Connected to a master server at 127.19.220.190:34233
I20250626 02:01:31.200599 21185 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:31.200794 21185 heartbeater.cc:507] Master 127.19.220.190:34233 requested a full tablet report, sending...
I20250626 02:01:31.201385 20905 ts_manager.cc:194] Registered new tserver with Master: 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:31.201439 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004617299s
I20250626 02:01:31.202026 20905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33476
I20250626 02:01:32.164188 21034 heartbeater.cc:499] Master 127.19.220.190:34233 was elected leader, sending a full tablet report...
I20250626 02:01:32.185099 21110 heartbeater.cc:499] Master 127.19.220.190:34233 was elected leader, sending a full tablet report...
I20250626 02:01:32.202826 21185 heartbeater.cc:499] Master 127.19.220.190:34233 was elected leader, sending a full tablet report...
I20250626 02:01:32.227866 20338 test_util.cc:276] Using random seed: -255528681
I20250626 02:01:32.235832 20905 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33490:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:01:32.236413 20905 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:01:32.248190 21149 tablet_service.cc:1468] Processing CreateTablet for tablet 12d287a85264480783ca51bf9cda9218 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250626 02:01:32.248548 21147 tablet_service.cc:1468] Processing CreateTablet for tablet 1a2f4557c0074123a846d6b8b2e42a27 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250626 02:01:32.248751 21149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d287a85264480783ca51bf9cda9218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.248971 21146 tablet_service.cc:1468] Processing CreateTablet for tablet fc511bed65b94bd7a4f024e0d56e92c9 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250626 02:01:32.249187 21146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc511bed65b94bd7a4f024e0d56e92c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.249641 20998 tablet_service.cc:1468] Processing CreateTablet for tablet 12d287a85264480783ca51bf9cda9218 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250626 02:01:32.249904 20998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d287a85264480783ca51bf9cda9218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.251269 21205 tablet_bootstrap.cc:492] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b: Bootstrap starting.
I20250626 02:01:32.251382 20993 tablet_service.cc:1468] Processing CreateTablet for tablet 5564fd2ee2be4984823c0cc317d82a36 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250626 02:01:32.251669 20993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5564fd2ee2be4984823c0cc317d82a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.251703 21144 tablet_service.cc:1468] Processing CreateTablet for tablet 5564fd2ee2be4984823c0cc317d82a36 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250626 02:01:32.251937 21144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5564fd2ee2be4984823c0cc317d82a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.252604 21205 tablet_bootstrap.cc:654] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.251423 21145 tablet_service.cc:1468] Processing CreateTablet for tablet fa76cd3c1c3c46ab981b3ba36b57b940 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250626 02:01:32.254262 21145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa76cd3c1c3c46ab981b3ba36b57b940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.254462 21147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2f4557c0074123a846d6b8b2e42a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.255087 20997 tablet_service.cc:1468] Processing CreateTablet for tablet f131b26b4fb844faab3e91abf00f6a65 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250626 02:01:32.255357 20997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f131b26b4fb844faab3e91abf00f6a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.256372 21206 tablet_bootstrap.cc:492] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715: Bootstrap starting.
I20250626 02:01:32.256361 21148 tablet_service.cc:1468] Processing CreateTablet for tablet f131b26b4fb844faab3e91abf00f6a65 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250626 02:01:32.256623 21148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f131b26b4fb844faab3e91abf00f6a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.257158 21073 tablet_service.cc:1468] Processing CreateTablet for tablet f131b26b4fb844faab3e91abf00f6a65 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250626 02:01:32.257308 21072 tablet_service.cc:1468] Processing CreateTablet for tablet 1a2f4557c0074123a846d6b8b2e42a27 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250626 02:01:32.257411 21206 tablet_bootstrap.cc:654] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.257422 21073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f131b26b4fb844faab3e91abf00f6a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.257553 21072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2f4557c0074123a846d6b8b2e42a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.257632 21074 tablet_service.cc:1468] Processing CreateTablet for tablet 12d287a85264480783ca51bf9cda9218 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250626 02:01:32.257850 21074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d287a85264480783ca51bf9cda9218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.259316 21209 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138: Bootstrap starting.
I20250626 02:01:32.259281 20995 tablet_service.cc:1468] Processing CreateTablet for tablet fc511bed65b94bd7a4f024e0d56e92c9 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250626 02:01:32.259529 20995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc511bed65b94bd7a4f024e0d56e92c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.259492 21143 tablet_service.cc:1468] Processing CreateTablet for tablet 4517caf93a944de796fe4d6e5f7e22cb (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250626 02:01:32.259711 21143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4517caf93a944de796fe4d6e5f7e22cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.259290 20992 tablet_service.cc:1468] Processing CreateTablet for tablet 4517caf93a944de796fe4d6e5f7e22cb (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250626 02:01:32.260293 21209 tablet_bootstrap.cc:654] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.260474 20992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4517caf93a944de796fe4d6e5f7e22cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.261662 21067 tablet_service.cc:1468] Processing CreateTablet for tablet be32c1f8be4f4267999798b3a7f83814 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250626 02:01:32.261646 20991 tablet_service.cc:1468] Processing CreateTablet for tablet be32c1f8be4f4267999798b3a7f83814 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250626 02:01:32.261916 21067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be32c1f8be4f4267999798b3a7f83814. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.261946 21209 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138: No bootstrap required, opened a new log
I20250626 02:01:32.261968 20991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be32c1f8be4f4267999798b3a7f83814. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.262010 21209 ts_tablet_manager.cc:1397] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:01:32.262478 21209 raft_consensus.cc:357] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.262583 21209 raft_consensus.cc:383] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.262626 21209 raft_consensus.cc:738] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.262750 21209 consensus_queue.cc:260] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.263135 21209 ts_tablet_manager.cc:1428] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.263288 21209 tablet_bootstrap.cc:492] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138: Bootstrap starting.
I20250626 02:01:32.263686 21142 tablet_service.cc:1468] Processing CreateTablet for tablet be32c1f8be4f4267999798b3a7f83814 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250626 02:01:32.263721 21070 tablet_service.cc:1468] Processing CreateTablet for tablet fa76cd3c1c3c46ab981b3ba36b57b940 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250626 02:01:32.263893 21071 tablet_service.cc:1468] Processing CreateTablet for tablet fc511bed65b94bd7a4f024e0d56e92c9 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250626 02:01:32.263980 21070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa76cd3c1c3c46ab981b3ba36b57b940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.264007 21069 tablet_service.cc:1468] Processing CreateTablet for tablet 5564fd2ee2be4984823c0cc317d82a36 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250626 02:01:32.264133 21071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc511bed65b94bd7a4f024e0d56e92c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.264364 21209 tablet_bootstrap.cc:654] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.263959 21142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be32c1f8be4f4267999798b3a7f83814. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.265368 21209 tablet_bootstrap.cc:492] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138: No bootstrap required, opened a new log
I20250626 02:01:32.265455 21209 ts_tablet_manager.cc:1397] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:32.265861 21209 raft_consensus.cc:357] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.265952 21209 raft_consensus.cc:383] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.265992 21209 raft_consensus.cc:738] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.266129 21209 consensus_queue.cc:260] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.266371 21209 ts_tablet_manager.cc:1428] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:32.266548 21209 tablet_bootstrap.cc:492] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138: Bootstrap starting.
I20250626 02:01:32.266508 21068 tablet_service.cc:1468] Processing CreateTablet for tablet 4517caf93a944de796fe4d6e5f7e22cb (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250626 02:01:32.266688 21069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5564fd2ee2be4984823c0cc317d82a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.266709 20994 tablet_service.cc:1468] Processing CreateTablet for tablet fa76cd3c1c3c46ab981b3ba36b57b940 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250626 02:01:32.266952 20994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa76cd3c1c3c46ab981b3ba36b57b940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.267494 21209 tablet_bootstrap.cc:654] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.255115 20996 tablet_service.cc:1468] Processing CreateTablet for tablet 1a2f4557c0074123a846d6b8b2e42a27 (DEFAULT_TABLE table=test-workload [id=42c51f51930841449f868c41f59b2d7a]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250626 02:01:32.268455 20996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2f4557c0074123a846d6b8b2e42a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.268595 21209 tablet_bootstrap.cc:492] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138: No bootstrap required, opened a new log
I20250626 02:01:32.268668 21209 ts_tablet_manager.cc:1397] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:32.269091 21209 raft_consensus.cc:357] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.269212 21209 raft_consensus.cc:383] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.269246 21209 raft_consensus.cc:738] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.269341 21209 consensus_queue.cc:260] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.269449 21068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4517caf93a944de796fe4d6e5f7e22cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:32.269585 21209 ts_tablet_manager.cc:1428] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.269755 21209 tablet_bootstrap.cc:492] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138: Bootstrap starting.
I20250626 02:01:32.270699 21209 tablet_bootstrap.cc:654] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.271889 21209 tablet_bootstrap.cc:492] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138: No bootstrap required, opened a new log
I20250626 02:01:32.271996 21209 ts_tablet_manager.cc:1397] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:32.272411 21209 raft_consensus.cc:357] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.272516 21209 raft_consensus.cc:383] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.272549 21209 raft_consensus.cc:738] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.272650 21209 consensus_queue.cc:260] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.272729 21206 tablet_bootstrap.cc:492] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715: No bootstrap required, opened a new log
I20250626 02:01:32.272789 21205 tablet_bootstrap.cc:492] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b: No bootstrap required, opened a new log
I20250626 02:01:32.272804 21206 ts_tablet_manager.cc:1397] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250626 02:01:32.272845 21205 ts_tablet_manager.cc:1397] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250626 02:01:32.272974 21209 ts_tablet_manager.cc:1428] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:32.273135 21209 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138: Bootstrap starting.
I20250626 02:01:32.273264 21205 raft_consensus.cc:357] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.273355 21205 raft_consensus.cc:383] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.273387 21205 raft_consensus.cc:738] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.273427 21206 raft_consensus.cc:357] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.273499 21206 raft_consensus.cc:383] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.273535 21206 raft_consensus.cc:738] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.273510 21205 consensus_queue.cc:260] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.273631 21206 consensus_queue.cc:260] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.273864 21205 ts_tablet_manager.cc:1428] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.273936 21206 ts_tablet_manager.cc:1428] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.274037 21205 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b: Bootstrap starting.
I20250626 02:01:32.274067 21206 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715: Bootstrap starting.
I20250626 02:01:32.274137 21209 tablet_bootstrap.cc:654] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.275024 21205 tablet_bootstrap.cc:654] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.275024 21206 tablet_bootstrap.cc:654] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.276281 21209 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138: No bootstrap required, opened a new log
I20250626 02:01:32.276391 21209 ts_tablet_manager.cc:1397] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:01:32.276655 21206 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715: No bootstrap required, opened a new log
I20250626 02:01:32.276787 21206 ts_tablet_manager.cc:1397] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:01:32.276800 21209 raft_consensus.cc:357] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.276935 21209 raft_consensus.cc:383] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.276986 21209 raft_consensus.cc:738] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.277122 21209 consensus_queue.cc:260] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.277294 21206 raft_consensus.cc:357] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.277395 21206 raft_consensus.cc:383] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.277436 21209 ts_tablet_manager.cc:1428] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:32.277443 21206 raft_consensus.cc:738] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.277626 21209 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138: Bootstrap starting.
I20250626 02:01:32.277618 21206 consensus_queue.cc:260] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.277923 21206 ts_tablet_manager.cc:1428] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.278119 21206 tablet_bootstrap.cc:492] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715: Bootstrap starting.
I20250626 02:01:32.278658 21209 tablet_bootstrap.cc:654] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.279083 21206 tablet_bootstrap.cc:654] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.280180 21206 tablet_bootstrap.cc:492] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715: No bootstrap required, opened a new log
I20250626 02:01:32.280278 21206 ts_tablet_manager.cc:1397] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:32.280524 21205 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b: No bootstrap required, opened a new log
I20250626 02:01:32.280606 21205 ts_tablet_manager.cc:1397] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250626 02:01:32.280719 21206 raft_consensus.cc:357] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.280840 21206 raft_consensus.cc:383] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.280882 21206 raft_consensus.cc:738] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.280932 21205 raft_consensus.cc:357] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.281021 21205 raft_consensus.cc:383] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.281049 21205 raft_consensus.cc:738] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.280995 21206 consensus_queue.cc:260] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.281157 21205 consensus_queue.cc:260] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.281421 21205 ts_tablet_manager.cc:1428] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.281580 21205 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b: Bootstrap starting.
I20250626 02:01:32.281821 21206 ts_tablet_manager.cc:1428] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.282003 21206 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715: Bootstrap starting.
I20250626 02:01:32.282474 21205 tablet_bootstrap.cc:654] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.283332 21206 tablet_bootstrap.cc:654] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.284489 21209 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138: No bootstrap required, opened a new log
I20250626 02:01:32.284598 21209 ts_tablet_manager.cc:1397] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250626 02:01:32.285096 21209 raft_consensus.cc:357] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.285199 21209 raft_consensus.cc:383] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.285234 21209 raft_consensus.cc:738] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.285247 21205 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b: No bootstrap required, opened a new log
I20250626 02:01:32.285322 21205 ts_tablet_manager.cc:1397] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250626 02:01:32.285362 21206 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715: No bootstrap required, opened a new log
I20250626 02:01:32.285355 21209 consensus_queue.cc:260] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.285466 21206 ts_tablet_manager.cc:1397] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:01:32.285627 21209 ts_tablet_manager.cc:1428] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:32.285694 21205 raft_consensus.cc:357] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.285780 21209 tablet_bootstrap.cc:492] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138: Bootstrap starting.
I20250626 02:01:32.285847 21206 raft_consensus.cc:357] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.285933 21206 raft_consensus.cc:383] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.285969 21206 raft_consensus.cc:738] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.286073 21206 consensus_queue.cc:260] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.285794 21205 raft_consensus.cc:383] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.286199 21205 raft_consensus.cc:738] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.286336 21205 consensus_queue.cc:260] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.286432 21206 ts_tablet_manager.cc:1428] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.286571 21206 tablet_bootstrap.cc:492] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715: Bootstrap starting.
I20250626 02:01:32.286610 21205 ts_tablet_manager.cc:1428] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.286792 21209 tablet_bootstrap.cc:654] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.286806 21205 tablet_bootstrap.cc:492] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b: Bootstrap starting.
I20250626 02:01:32.287555 21206 tablet_bootstrap.cc:654] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.287773 21205 tablet_bootstrap.cc:654] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.288627 21209 tablet_bootstrap.cc:492] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138: No bootstrap required, opened a new log
I20250626 02:01:32.288722 21209 ts_tablet_manager.cc:1397] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250626 02:01:32.288985 21205 tablet_bootstrap.cc:492] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b: No bootstrap required, opened a new log
I20250626 02:01:32.289088 21205 ts_tablet_manager.cc:1397] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:01:32.289116 21209 raft_consensus.cc:357] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.289219 21209 raft_consensus.cc:383] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.289260 21209 raft_consensus.cc:738] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.289374 21209 consensus_queue.cc:260] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.289541 21205 raft_consensus.cc:357] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.289650 21205 raft_consensus.cc:383] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.289696 21205 raft_consensus.cc:738] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.289827 21205 consensus_queue.cc:260] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.289997 21206 tablet_bootstrap.cc:492] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715: No bootstrap required, opened a new log
I20250626 02:01:32.290077 21206 ts_tablet_manager.cc:1397] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250626 02:01:32.290174 21205 ts_tablet_manager.cc:1428] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:32.290366 21205 tablet_bootstrap.cc:492] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b: Bootstrap starting.
I20250626 02:01:32.290514 21206 raft_consensus.cc:357] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.290619 21206 raft_consensus.cc:383] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.290660 21206 raft_consensus.cc:738] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.290805 21206 consensus_queue.cc:260] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.291343 21205 tablet_bootstrap.cc:654] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.291623 21206 ts_tablet_manager.cc:1428] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.291802 21206 tablet_bootstrap.cc:492] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715: Bootstrap starting.
I20250626 02:01:32.289601 21209 ts_tablet_manager.cc:1428] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:32.292001 21209 tablet_bootstrap.cc:492] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138: Bootstrap starting.
I20250626 02:01:32.292815 21206 tablet_bootstrap.cc:654] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.292953 21209 tablet_bootstrap.cc:654] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.293859 21206 tablet_bootstrap.cc:492] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715: No bootstrap required, opened a new log
I20250626 02:01:32.293949 21206 ts_tablet_manager.cc:1397] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:32.294039 21209 tablet_bootstrap.cc:492] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138: No bootstrap required, opened a new log
I20250626 02:01:32.294129 21209 ts_tablet_manager.cc:1397] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:01:32.294373 21206 raft_consensus.cc:357] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.294472 21206 raft_consensus.cc:383] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.294513 21206 raft_consensus.cc:738] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.294634 21206 consensus_queue.cc:260] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.294633 21209 raft_consensus.cc:357] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.294780 21209 raft_consensus.cc:383] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.294921 21206 ts_tablet_manager.cc:1428] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.294909 21209 raft_consensus.cc:738] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.295132 21206 tablet_bootstrap.cc:492] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715: Bootstrap starting.
I20250626 02:01:32.295090 21209 consensus_queue.cc:260] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.295612 21205 tablet_bootstrap.cc:492] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b: No bootstrap required, opened a new log
I20250626 02:01:32.295684 21205 ts_tablet_manager.cc:1397] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250626 02:01:32.296094 21206 tablet_bootstrap.cc:654] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.296327 21209 ts_tablet_manager.cc:1428] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:01:32.296077 21205 raft_consensus.cc:357] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.296519 21205 raft_consensus.cc:383] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.296555 21205 raft_consensus.cc:738] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.296676 21205 consensus_queue.cc:260] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.297230 21205 ts_tablet_manager.cc:1428] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:01:32.297405 21205 tablet_bootstrap.cc:492] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b: Bootstrap starting.
I20250626 02:01:32.297926 21206 tablet_bootstrap.cc:492] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715: No bootstrap required, opened a new log
I20250626 02:01:32.298007 21206 ts_tablet_manager.cc:1397] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:01:32.298337 21206 raft_consensus.cc:357] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.298384 21205 tablet_bootstrap.cc:654] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.298429 21206 raft_consensus.cc:383] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.298470 21206 raft_consensus.cc:738] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.298591 21206 consensus_queue.cc:260] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.298800 21206 ts_tablet_manager.cc:1428] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:32.298949 21206 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715: Bootstrap starting.
I20250626 02:01:32.299459 21205 tablet_bootstrap.cc:492] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b: No bootstrap required, opened a new log
I20250626 02:01:32.299535 21205 ts_tablet_manager.cc:1397] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:01:32.299962 21205 raft_consensus.cc:357] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.300057 21205 raft_consensus.cc:383] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.300195 21205 raft_consensus.cc:738] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.300359 21206 tablet_bootstrap.cc:654] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.300324 21205 consensus_queue.cc:260] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.300987 21205 ts_tablet_manager.cc:1428] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:32.301177 21205 tablet_bootstrap.cc:492] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b: Bootstrap starting.
I20250626 02:01:32.301748 21206 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715: No bootstrap required, opened a new log
I20250626 02:01:32.301832 21206 ts_tablet_manager.cc:1397] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:01:32.302057 21205 tablet_bootstrap.cc:654] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.302260 21206 raft_consensus.cc:357] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.302371 21206 raft_consensus.cc:383] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.302409 21206 raft_consensus.cc:738] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.302510 21206 consensus_queue.cc:260] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.302779 21206 ts_tablet_manager.cc:1428] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:32.303156 21205 tablet_bootstrap.cc:492] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b: No bootstrap required, opened a new log
I20250626 02:01:32.303241 21205 ts_tablet_manager.cc:1397] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250626 02:01:32.303577 21211 raft_consensus.cc:491] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.303650 21205 raft_consensus.cc:357] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.303676 21211 raft_consensus.cc:513] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.303745 21205 raft_consensus.cc:383] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.303772 21205 raft_consensus.cc:738] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.303882 21205 consensus_queue.cc:260] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.304121 21205 ts_tablet_manager.cc:1428] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:32.304327 21205 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b: Bootstrap starting.
I20250626 02:01:32.304348 21211 leader_election.cc:290] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.305341 21205 tablet_bootstrap.cc:654] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:32.306629 21205 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b: No bootstrap required, opened a new log
I20250626 02:01:32.306702 21205 ts_tablet_manager.cc:1397] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:32.306923 21211 raft_consensus.cc:491] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.307009 21211 raft_consensus.cc:513] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.307143 21205 raft_consensus.cc:357] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.307245 21205 raft_consensus.cc:383] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:32.307286 21205 raft_consensus.cc:738] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Initialized, Role: FOLLOWER
I20250626 02:01:32.307348 21211 leader_election.cc:290] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.307427 21205 consensus_queue.cc:260] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.307701 21205 ts_tablet_manager.cc:1428] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.307754 21216 raft_consensus.cc:491] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.307835 21216 raft_consensus.cc:513] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.308223 21216 leader_election.cc:290] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.309799 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a2f4557c0074123a846d6b8b2e42a27" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b" is_pre_election: true
I20250626 02:01:32.309899 21159 raft_consensus.cc:2466] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.309993 21008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a2f4557c0074123a846d6b8b2e42a27" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715" is_pre_election: true
I20250626 02:01:32.310109 21158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa76cd3c1c3c46ab981b3ba36b57b940" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b" is_pre_election: true
I20250626 02:01:32.310202 21158 raft_consensus.cc:2466] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.310094 21008 raft_consensus.cc:2466] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.310096 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa76cd3c1c3c46ab981b3ba36b57b940" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715" is_pre_election: true
I20250626 02:01:32.310495 21007 raft_consensus.cc:2466] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.310597 21047 leader_election.cc:304] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b029e769a34e469c6bc80ad083f28b, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.310736 21211 raft_consensus.cc:2802] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:32.310796 21211 raft_consensus.cc:491] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:32.310838 21211 raft_consensus.cc:3058] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.310817 21047 leader_election.cc:304] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b029e769a34e469c6bc80ad083f28b, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.311676 21211 raft_consensus.cc:513] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.311969 21211 leader_election.cc:290] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Requested vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.312086 21211 raft_consensus.cc:2802] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:32.312160 21211 raft_consensus.cc:491] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:32.312208 21211 raft_consensus.cc:3058] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.312276 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a2f4557c0074123a846d6b8b2e42a27" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
I20250626 02:01:32.312379 21007 raft_consensus.cc:3058] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.312830 21211 raft_consensus.cc:513] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.313097 21211 leader_election.cc:290] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Requested vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.313205 21007 raft_consensus.cc:2466] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.313246 21158 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a2f4557c0074123a846d6b8b2e42a27" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b"
I20250626 02:01:32.313333 21158 raft_consensus.cc:3058] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.313444 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa76cd3c1c3c46ab981b3ba36b57b940" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b"
I20250626 02:01:32.313555 21159 raft_consensus.cc:3058] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.313570 21050 leader_election.cc:304] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.313720 21211 raft_consensus.cc:2802] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:32.313794 21211 raft_consensus.cc:695] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Becoming Leader. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Running, Role: LEADER
I20250626 02:01:32.313815 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa76cd3c1c3c46ab981b3ba36b57b940" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
I20250626 02:01:32.313898 21007 raft_consensus.cc:3058] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.313912 21211 consensus_queue.cc:237] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.314124 21158 raft_consensus.cc:2466] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.314481 21159 raft_consensus.cc:2466] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.314688 21007 raft_consensus.cc:2466] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.314874 21047 leader_election.cc:304] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b029e769a34e469c6bc80ad083f28b, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.315050 21211 raft_consensus.cc:2802] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:32.315096 21227 raft_consensus.cc:491] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.315124 21211 raft_consensus.cc:695] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Becoming Leader. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Running, Role: LEADER
I20250626 02:01:32.315176 21227 raft_consensus.cc:513] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.315263 21211 consensus_queue.cc:237] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.315789 20903 catalog_manager.cc:5582] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 reported cstate change: term changed from 0 to 1, leader changed from <none> to c089d4f3d49d4a24a66478348d576138 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "c089d4f3d49d4a24a66478348d576138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:32.317027 21227 leader_election.cc:290] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001)
I20250626 02:01:32.319375 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa76cd3c1c3c46ab981b3ba36b57b940" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c089d4f3d49d4a24a66478348d576138" is_pre_election: true
I20250626 02:01:32.319398 20903 catalog_manager.cc:5582] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 reported cstate change: term changed from 0 to 1, leader changed from <none> to c089d4f3d49d4a24a66478348d576138 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "c089d4f3d49d4a24a66478348d576138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:32.320927 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa76cd3c1c3c46ab981b3ba36b57b940" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b" is_pre_election: true
I20250626 02:01:32.321085 21159 raft_consensus.cc:2391] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2afdb4ad450b40fbba9bf06deaf50715 in current term 1: Already voted for candidate c089d4f3d49d4a24a66478348d576138 in this term.
I20250626 02:01:32.321369 20971 leader_election.cc:304] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715; no voters: 93b029e769a34e469c6bc80ad083f28b, c089d4f3d49d4a24a66478348d576138
I20250626 02:01:32.321516 21216 raft_consensus.cc:2747] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 02:01:32.322633 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f131b26b4fb844faab3e91abf00f6a65" candidate_uuid: "93b029e769a34e469c6bc80ad083f28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715" is_pre_election: true
I20250626 02:01:32.322746 21007 raft_consensus.cc:2466] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93b029e769a34e469c6bc80ad083f28b in term 0.
I20250626 02:01:32.322999 21125 leader_election.cc:304] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, 93b029e769a34e469c6bc80ad083f28b; no voters:
I20250626 02:01:32.323168 21227 raft_consensus.cc:2802] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:32.323232 21227 raft_consensus.cc:491] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:32.323284 21227 raft_consensus.cc:3058] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.323395 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f131b26b4fb844faab3e91abf00f6a65" candidate_uuid: "93b029e769a34e469c6bc80ad083f28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c089d4f3d49d4a24a66478348d576138" is_pre_election: true
I20250626 02:01:32.323505 21084 raft_consensus.cc:2466] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93b029e769a34e469c6bc80ad083f28b in term 0.
I20250626 02:01:32.324100 21227 raft_consensus.cc:513] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.324467 21227 leader_election.cc:290] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [CANDIDATE]: Term 1 election: Requested vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001)
I20250626 02:01:32.324755 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f131b26b4fb844faab3e91abf00f6a65" candidate_uuid: "93b029e769a34e469c6bc80ad083f28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
I20250626 02:01:32.324769 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f131b26b4fb844faab3e91abf00f6a65" candidate_uuid: "93b029e769a34e469c6bc80ad083f28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c089d4f3d49d4a24a66478348d576138"
I20250626 02:01:32.324846 21007 raft_consensus.cc:3058] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.324879 21084 raft_consensus.cc:3058] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.325508 21007 raft_consensus.cc:2466] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93b029e769a34e469c6bc80ad083f28b in term 1.
I20250626 02:01:32.325646 21084 raft_consensus.cc:2466] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93b029e769a34e469c6bc80ad083f28b in term 1.
I20250626 02:01:32.325771 21125 leader_election.cc:304] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, 93b029e769a34e469c6bc80ad083f28b; no voters:
I20250626 02:01:32.325939 21227 raft_consensus.cc:2802] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:32.326035 21227 raft_consensus.cc:695] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 1 LEADER]: Becoming Leader. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Running, Role: LEADER
I20250626 02:01:32.326164 21227 consensus_queue.cc:237] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.327572 20903 catalog_manager.cc:5582] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b reported cstate change: term changed from 0 to 1, leader changed from <none> to 93b029e769a34e469c6bc80ad083f28b (127.19.220.131). New cstate: current_term: 1 leader_uuid: "93b029e769a34e469c6bc80ad083f28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: HEALTHY } } }
W20250626 02:01:32.407138 21035 tablet.cc:2378] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 02:01:32.407986 21241 raft_consensus.cc:491] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.408114 21241 raft_consensus.cc:513] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.408674 21241 leader_election.cc:290] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.409070 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12d287a85264480783ca51bf9cda9218" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715" is_pre_election: true
I20250626 02:01:32.409190 21007 raft_consensus.cc:2466] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.409184 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12d287a85264480783ca51bf9cda9218" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b" is_pre_election: true
I20250626 02:01:32.409346 21159 raft_consensus.cc:2466] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.409569 21050 leader_election.cc:304] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.409787 21241 raft_consensus.cc:2802] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:32.409850 21241 raft_consensus.cc:491] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:32.409894 21241 raft_consensus.cc:3058] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.411087 21241 raft_consensus.cc:513] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.411477 21241 leader_election.cc:290] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Requested vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.411818 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12d287a85264480783ca51bf9cda9218" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b"
I20250626 02:01:32.411854 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12d287a85264480783ca51bf9cda9218" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
I20250626 02:01:32.411921 21159 raft_consensus.cc:3058] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.411983 21007 raft_consensus.cc:3058] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.412883 21159 raft_consensus.cc:2466] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.412997 21007 raft_consensus.cc:2466] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.413165 21047 leader_election.cc:304] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b029e769a34e469c6bc80ad083f28b, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.413414 21241 raft_consensus.cc:2802] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:32.413590 21241 raft_consensus.cc:695] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Becoming Leader. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Running, Role: LEADER
I20250626 02:01:32.413726 21241 consensus_queue.cc:237] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.415506 20903 catalog_manager.cc:5582] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 reported cstate change: term changed from 0 to 1, leader changed from <none> to c089d4f3d49d4a24a66478348d576138 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "c089d4f3d49d4a24a66478348d576138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:32.446511 21241 raft_consensus.cc:491] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.446659 21241 raft_consensus.cc:513] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.447026 21241 leader_election.cc:290] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.447431 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc511bed65b94bd7a4f024e0d56e92c9" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b" is_pre_election: true
I20250626 02:01:32.447438 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc511bed65b94bd7a4f024e0d56e92c9" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715" is_pre_election: true
I20250626 02:01:32.447575 21159 raft_consensus.cc:2466] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.447621 21007 raft_consensus.cc:2466] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.447860 21050 leader_election.cc:304] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.448036 21241 raft_consensus.cc:2802] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:32.448096 21241 raft_consensus.cc:491] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:32.448181 21241 raft_consensus.cc:3058] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.449136 21241 raft_consensus.cc:513] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.449458 21241 leader_election.cc:290] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Requested vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.449802 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc511bed65b94bd7a4f024e0d56e92c9" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
I20250626 02:01:32.449820 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc511bed65b94bd7a4f024e0d56e92c9" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b"
I20250626 02:01:32.449915 21159 raft_consensus.cc:3058] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.449918 21007 raft_consensus.cc:3058] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.450656 21159 raft_consensus.cc:2466] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.450659 21007 raft_consensus.cc:2466] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.450922 21047 leader_election.cc:304] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b029e769a34e469c6bc80ad083f28b, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.451115 21241 raft_consensus.cc:2802] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:32.451189 21241 raft_consensus.cc:695] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Becoming Leader. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Running, Role: LEADER
I20250626 02:01:32.451335 21241 consensus_queue.cc:237] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.452883 20903 catalog_manager.cc:5582] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 reported cstate change: term changed from 0 to 1, leader changed from <none> to c089d4f3d49d4a24a66478348d576138 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "c089d4f3d49d4a24a66478348d576138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:32.481204 21241 raft_consensus.cc:491] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.481365 21241 raft_consensus.cc:513] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.481823 21241 leader_election.cc:290] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.482273 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4517caf93a944de796fe4d6e5f7e22cb" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715" is_pre_election: true
I20250626 02:01:32.482270 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4517caf93a944de796fe4d6e5f7e22cb" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b" is_pre_election: true
I20250626 02:01:32.482416 21007 raft_consensus.cc:2466] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.482451 21159 raft_consensus.cc:2466] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 0.
I20250626 02:01:32.482728 21050 leader_election.cc:304] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.482889 21241 raft_consensus.cc:2802] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:32.482951 21241 raft_consensus.cc:491] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:32.482980 21241 raft_consensus.cc:3058] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.483817 21241 raft_consensus.cc:513] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.484172 21241 leader_election.cc:290] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Requested vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.484519 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4517caf93a944de796fe4d6e5f7e22cb" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b"
I20250626 02:01:32.484519 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4517caf93a944de796fe4d6e5f7e22cb" candidate_uuid: "c089d4f3d49d4a24a66478348d576138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
I20250626 02:01:32.484620 21159 raft_consensus.cc:3058] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.484656 21007 raft_consensus.cc:3058] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.485383 21007 raft_consensus.cc:2466] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.485386 21159 raft_consensus.cc:2466] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c089d4f3d49d4a24a66478348d576138 in term 1.
I20250626 02:01:32.485685 21047 leader_election.cc:304] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b029e769a34e469c6bc80ad083f28b, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.485859 21241 raft_consensus.cc:2802] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:32.485960 21241 raft_consensus.cc:695] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Becoming Leader. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Running, Role: LEADER
I20250626 02:01:32.486099 21241 consensus_queue.cc:237] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.487618 20903 catalog_manager.cc:5582] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 reported cstate change: term changed from 0 to 1, leader changed from <none> to c089d4f3d49d4a24a66478348d576138 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "c089d4f3d49d4a24a66478348d576138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:32.508095 21227 raft_consensus.cc:491] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.508281 21227 raft_consensus.cc:513] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.508694 21227 leader_election.cc:290] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001)
I20250626 02:01:32.509157 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5564fd2ee2be4984823c0cc317d82a36" candidate_uuid: "93b029e769a34e469c6bc80ad083f28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c089d4f3d49d4a24a66478348d576138" is_pre_election: true
I20250626 02:01:32.509153 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5564fd2ee2be4984823c0cc317d82a36" candidate_uuid: "93b029e769a34e469c6bc80ad083f28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715" is_pre_election: true
I20250626 02:01:32.509299 21084 raft_consensus.cc:2466] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93b029e769a34e469c6bc80ad083f28b in term 0.
I20250626 02:01:32.509325 21007 raft_consensus.cc:2466] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93b029e769a34e469c6bc80ad083f28b in term 0.
I20250626 02:01:32.509613 21125 leader_election.cc:304] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, 93b029e769a34e469c6bc80ad083f28b; no voters:
I20250626 02:01:32.509769 21227 raft_consensus.cc:2802] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:32.509825 21227 raft_consensus.cc:491] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:32.509855 21227 raft_consensus.cc:3058] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.511013 21227 raft_consensus.cc:513] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.511348 21227 leader_election.cc:290] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [CANDIDATE]: Term 1 election: Requested vote from peers 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863), c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001)
I20250626 02:01:32.511677 21007 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5564fd2ee2be4984823c0cc317d82a36" candidate_uuid: "93b029e769a34e469c6bc80ad083f28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
I20250626 02:01:32.511698 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5564fd2ee2be4984823c0cc317d82a36" candidate_uuid: "93b029e769a34e469c6bc80ad083f28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c089d4f3d49d4a24a66478348d576138"
I20250626 02:01:32.511759 21007 raft_consensus.cc:3058] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.511797 21084 raft_consensus.cc:3058] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.512511 21007 raft_consensus.cc:2466] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93b029e769a34e469c6bc80ad083f28b in term 1.
I20250626 02:01:32.512665 21084 raft_consensus.cc:2466] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93b029e769a34e469c6bc80ad083f28b in term 1.
I20250626 02:01:32.512777 21125 leader_election.cc:304] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, 93b029e769a34e469c6bc80ad083f28b; no voters:
I20250626 02:01:32.512950 21227 raft_consensus.cc:2802] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:32.513029 21227 raft_consensus.cc:695] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 1 LEADER]: Becoming Leader. State: Replica: 93b029e769a34e469c6bc80ad083f28b, State: Running, Role: LEADER
I20250626 02:01:32.513173 21227 consensus_queue.cc:237] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.514671 20903 catalog_manager.cc:5582] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b reported cstate change: term changed from 0 to 1, leader changed from <none> to 93b029e769a34e469c6bc80ad083f28b (127.19.220.131). New cstate: current_term: 1 leader_uuid: "93b029e769a34e469c6bc80ad083f28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: HEALTHY } } }
I20250626 02:01:32.593581 21216 raft_consensus.cc:491] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:32.593732 21216 raft_consensus.cc:513] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.594195 21216 leader_election.cc:290] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.594631 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be32c1f8be4f4267999798b3a7f83814" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c089d4f3d49d4a24a66478348d576138" is_pre_election: true
I20250626 02:01:32.594650 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be32c1f8be4f4267999798b3a7f83814" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b" is_pre_election: true
I20250626 02:01:32.594750 21084 raft_consensus.cc:2466] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2afdb4ad450b40fbba9bf06deaf50715 in term 0.
I20250626 02:01:32.594799 21159 raft_consensus.cc:2466] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2afdb4ad450b40fbba9bf06deaf50715 in term 0.
I20250626 02:01:32.595023 20971 leader_election.cc:304] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, c089d4f3d49d4a24a66478348d576138; no voters:
I20250626 02:01:32.595180 21216 raft_consensus.cc:2802] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:32.595229 21216 raft_consensus.cc:491] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:32.595255 21216 raft_consensus.cc:3058] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.596163 21216 raft_consensus.cc:513] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.596448 21216 leader_election.cc:290] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 1 election: Requested vote from peers c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:32.596801 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be32c1f8be4f4267999798b3a7f83814" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c089d4f3d49d4a24a66478348d576138"
I20250626 02:01:32.596801 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be32c1f8be4f4267999798b3a7f83814" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b029e769a34e469c6bc80ad083f28b"
I20250626 02:01:32.596939 21084 raft_consensus.cc:3058] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.596968 21159 raft_consensus.cc:3058] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:32.597805 21159 raft_consensus.cc:2466] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2afdb4ad450b40fbba9bf06deaf50715 in term 1.
I20250626 02:01:32.597805 21084 raft_consensus.cc:2466] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2afdb4ad450b40fbba9bf06deaf50715 in term 1.
I20250626 02:01:32.598084 20971 leader_election.cc:304] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, 93b029e769a34e469c6bc80ad083f28b; no voters:
I20250626 02:01:32.598222 21216 raft_consensus.cc:2802] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:32.598423 21216 raft_consensus.cc:695] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 LEADER]: Becoming Leader. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Running, Role: LEADER
I20250626 02:01:32.598587 21216 consensus_queue.cc:237] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:32.599941 20903 catalog_manager.cc:5582] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "2afdb4ad450b40fbba9bf06deaf50715" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:32.613816 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:32.615579 21255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:32.615579 21254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:32.615620 21257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:32.615790 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:32.616041 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:32.616087 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:32.616103 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903292616103 us; error 0 us; skew 500 ppm
I20250626 02:01:32.617179 20338 webserver.cc:469] Webserver started at http://127.19.220.132:46373/ using document root <none> and password file <none>
I20250626 02:01:32.617316 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:32.617362 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:32.617442 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:32.617794 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-3-root/instance:
uuid: "8ee709b1b6bd409ba38922118e63f539"
format_stamp: "Formatted at 2025-06-26 02:01:32 on dist-test-slave-h18f"
I20250626 02:01:32.619148 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:01:32.619994 21262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:32.620242 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:32.620309 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-3-root
uuid: "8ee709b1b6bd409ba38922118e63f539"
format_stamp: "Formatted at 2025-06-26 02:01:32 on dist-test-slave-h18f"
I20250626 02:01:32.620383 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750903288526542-20338-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:32.629873 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:32.630295 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:32.630579 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:32.631019 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:32.631060 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:32.631109 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:32.631140 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:32.636283 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.132:45955
I20250626 02:01:32.636327 21325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.132:45955 every 8 connection(s)
I20250626 02:01:32.640453 21326 heartbeater.cc:344] Connected to a master server at 127.19.220.190:34233
I20250626 02:01:32.640547 21326 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:32.640754 21326 heartbeater.cc:507] Master 127.19.220.190:34233 requested a full tablet report, sending...
I20250626 02:01:32.641335 20903 ts_manager.cc:194] Registered new tserver with Master: 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132:45955)
I20250626 02:01:32.641847 20903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37380
I20250626 02:01:32.707861 21241 consensus_queue.cc:1035] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.710940 21241 consensus_queue.cc:1035] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.731226 21241 consensus_queue.cc:1035] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.734254 21241 consensus_queue.cc:1035] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.768325 21227 consensus_queue.cc:1035] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.771112 21227 consensus_queue.cc:1035] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.829216 21241 consensus_queue.cc:1035] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.832065 21339 consensus_queue.cc:1035] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.900946 21211 consensus_queue.cc:1035] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.903753 21339 consensus_queue.cc:1035] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.914404 21227 consensus_queue.cc:1035] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.917490 21227 consensus_queue.cc:1035] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.945942 21339 consensus_queue.cc:1035] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:32.949477 21241 consensus_queue.cc:1035] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:33.106215 20959 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:33.115471 21084 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "12d287a85264480783ca51bf9cda9218"
dest_uuid: "c089d4f3d49d4a24a66478348d576138"
mode: GRACEFUL
new_leader_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
from {username='slave'} at 127.0.0.1:36374
I20250626 02:01:33.115654 21084 raft_consensus.cc:604] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Received request to transfer leadership to TS 2afdb4ad450b40fbba9bf06deaf50715
I20250626 02:01:33.116477 21084 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fc511bed65b94bd7a4f024e0d56e92c9"
dest_uuid: "c089d4f3d49d4a24a66478348d576138"
mode: GRACEFUL
new_leader_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
from {username='slave'} at 127.0.0.1:36374
I20250626 02:01:33.116587 21084 raft_consensus.cc:604] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Received request to transfer leadership to TS 2afdb4ad450b40fbba9bf06deaf50715
I20250626 02:01:33.116870 20959 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250626 02:01:33.117097 20959 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:33.180645 21216 consensus_queue.cc:1035] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:33.183585 21216 consensus_queue.cc:1035] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:33.442348 21241 raft_consensus.cc:991] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138: : Instructing follower 2afdb4ad450b40fbba9bf06deaf50715 to start an election
I20250626 02:01:33.442515 21241 raft_consensus.cc:1079] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Signalling peer 2afdb4ad450b40fbba9bf06deaf50715 to start an election
I20250626 02:01:33.443112 21007 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "12d287a85264480783ca51bf9cda9218"
dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
from {username='slave'} at 127.0.0.1:56136
I20250626 02:01:33.443249 21007 raft_consensus.cc:491] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:01:33.443298 21007 raft_consensus.cc:3058] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:33.444080 21007 raft_consensus.cc:513] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:33.444445 21007 leader_election.cc:290] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 2 election: Requested vote from peers c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:33.444762 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12d287a85264480783ca51bf9cda9218" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c089d4f3d49d4a24a66478348d576138"
I20250626 02:01:33.444835 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12d287a85264480783ca51bf9cda9218" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93b029e769a34e469c6bc80ad083f28b"
I20250626 02:01:33.444883 21084 raft_consensus.cc:3053] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:01:33.444932 21084 raft_consensus.cc:738] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Running, Role: LEADER
I20250626 02:01:33.444949 21159 raft_consensus.cc:3058] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:33.445093 21084 consensus_queue.cc:260] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:33.445245 21084 raft_consensus.cc:3058] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:33.445833 21159 raft_consensus.cc:2466] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2afdb4ad450b40fbba9bf06deaf50715 in term 2.
I20250626 02:01:33.446094 21084 raft_consensus.cc:2466] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2afdb4ad450b40fbba9bf06deaf50715 in term 2.
I20250626 02:01:33.446146 20971 leader_election.cc:304] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, 93b029e769a34e469c6bc80ad083f28b; no voters:
I20250626 02:01:33.446307 21216 raft_consensus.cc:2802] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:01:33.446393 21216 raft_consensus.cc:695] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 LEADER]: Becoming Leader. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Running, Role: LEADER
I20250626 02:01:33.446534 21216 consensus_queue.cc:237] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:33.448338 20903 catalog_manager.cc:5582] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 reported cstate change: term changed from 1 to 2, leader changed from c089d4f3d49d4a24a66478348d576138 (127.19.220.130) to 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129). New cstate: current_term: 2 leader_uuid: "2afdb4ad450b40fbba9bf06deaf50715" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:33.451140 21241 raft_consensus.cc:991] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138: : Instructing follower 2afdb4ad450b40fbba9bf06deaf50715 to start an election
I20250626 02:01:33.451459 21241 raft_consensus.cc:1079] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Signalling peer 2afdb4ad450b40fbba9bf06deaf50715 to start an election
I20250626 02:01:33.451835 21007 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fc511bed65b94bd7a4f024e0d56e92c9"
dest_uuid: "2afdb4ad450b40fbba9bf06deaf50715"
from {username='slave'} at 127.0.0.1:56136
I20250626 02:01:33.451951 21007 raft_consensus.cc:491] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:01:33.452020 21007 raft_consensus.cc:3058] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:33.452793 21007 raft_consensus.cc:513] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:33.453116 21007 leader_election.cc:290] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 2 election: Requested vote from peers c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001), 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:33.453496 21084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc511bed65b94bd7a4f024e0d56e92c9" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c089d4f3d49d4a24a66478348d576138"
I20250626 02:01:33.453536 21159 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc511bed65b94bd7a4f024e0d56e92c9" candidate_uuid: "2afdb4ad450b40fbba9bf06deaf50715" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "93b029e769a34e469c6bc80ad083f28b"
I20250626 02:01:33.453605 21084 raft_consensus.cc:3053] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:01:33.453605 21159 raft_consensus.cc:3058] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:33.453672 21084 raft_consensus.cc:738] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c089d4f3d49d4a24a66478348d576138, State: Running, Role: LEADER
I20250626 02:01:33.453817 21084 consensus_queue.cc:260] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:33.454010 21084 raft_consensus.cc:3058] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:33.454600 21159 raft_consensus.cc:2466] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2afdb4ad450b40fbba9bf06deaf50715 in term 2.
I20250626 02:01:33.454874 20971 leader_election.cc:304] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afdb4ad450b40fbba9bf06deaf50715, 93b029e769a34e469c6bc80ad083f28b; no voters:
I20250626 02:01:33.455032 21216 raft_consensus.cc:2802] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:01:33.455060 21084 raft_consensus.cc:2466] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2afdb4ad450b40fbba9bf06deaf50715 in term 2.
I20250626 02:01:33.455111 21216 raft_consensus.cc:695] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 LEADER]: Becoming Leader. State: Replica: 2afdb4ad450b40fbba9bf06deaf50715, State: Running, Role: LEADER
I20250626 02:01:33.455286 21216 consensus_queue.cc:237] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } }
I20250626 02:01:33.456566 20904 catalog_manager.cc:5582] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 reported cstate change: term changed from 1 to 2, leader changed from c089d4f3d49d4a24a66478348d576138 (127.19.220.130) to 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129). New cstate: current_term: 2 leader_uuid: "2afdb4ad450b40fbba9bf06deaf50715" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:33.642704 21326 heartbeater.cc:499] Master 127.19.220.190:34233 was elected leader, sending a full tablet report...
I20250626 02:01:33.852389 21159 raft_consensus.cc:1273] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Refusing update from remote peer 2afdb4ad450b40fbba9bf06deaf50715: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:01:33.852828 21356 consensus_queue.cc:1035] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:33.855051 21084 raft_consensus.cc:1273] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Refusing update from remote peer 2afdb4ad450b40fbba9bf06deaf50715: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:01:33.855629 21216 consensus_queue.cc:1035] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:33.909401 21084 raft_consensus.cc:1273] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Refusing update from remote peer 2afdb4ad450b40fbba9bf06deaf50715: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:01:33.909823 21216 consensus_queue.cc:1035] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:33.912252 21159 raft_consensus.cc:1273] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Refusing update from remote peer 2afdb4ad450b40fbba9bf06deaf50715: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:01:33.912770 21350 consensus_queue.cc:1035] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:35.117555 20959 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:35.118345 20959 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:35.118407 20959 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:37.118763 20959 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:37.119527 20959 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:37.119590 20959 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:38.114579 21007 consensus_queue.cc:237] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } attrs { replace: true } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.116212 21084 raft_consensus.cc:1273] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Refusing update from remote peer 2afdb4ad450b40fbba9bf06deaf50715: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 02:01:38.116326 21159 raft_consensus.cc:1273] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Refusing update from remote peer 2afdb4ad450b40fbba9bf06deaf50715: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 02:01:38.116636 21364 consensus_queue.cc:1035] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:01:38.116815 21370 consensus_queue.cc:1035] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:01:38.118693 21369 raft_consensus.cc:2953] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } attrs { replace: true } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.119160 21084 raft_consensus.cc:2953] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } attrs { replace: true } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.119290 21159 raft_consensus.cc:2953] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } attrs { replace: true } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
W20250626 02:01:38.121719 20974 consensus_peers.cc:487] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 -> Peer 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132:45955): Couldn't send request to peer 8ee709b1b6bd409ba38922118e63f539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc511bed65b94bd7a4f024e0d56e92c9. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:01:38.121901 20904 catalog_manager.cc:5582] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 reported cstate change: config changed from index -1 to 3, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New cstate: current_term: 2 leader_uuid: "2afdb4ad450b40fbba9bf06deaf50715" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:38.129287 21084 consensus_queue.cc:237] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } attrs { replace: true } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.130640 21159 raft_consensus.cc:1273] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Refusing update from remote peer c089d4f3d49d4a24a66478348d576138: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:01:38.130932 21007 raft_consensus.cc:1273] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Refusing update from remote peer c089d4f3d49d4a24a66478348d576138: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:01:38.130985 21382 consensus_queue.cc:1035] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:38.131286 21241 consensus_queue.cc:1035] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:38.132568 21241 raft_consensus.cc:2953] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } attrs { replace: true } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.133019 21159 raft_consensus.cc:2953] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } attrs { replace: true } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.134874 20905 catalog_manager.cc:5582] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 reported cstate change: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "c089d4f3d49d4a24a66478348d576138" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:38.135004 21007 raft_consensus.cc:2953] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } attrs { replace: true } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
W20250626 02:01:38.136004 21050 consensus_peers.cc:487] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 -> Peer 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132:45955): Couldn't send request to peer 8ee709b1b6bd409ba38922118e63f539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a2f4557c0074123a846d6b8b2e42a27. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:01:38.140254 21159 consensus_queue.cc:237] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.141638 21084 raft_consensus.cc:1273] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Refusing update from remote peer 93b029e769a34e469c6bc80ad083f28b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:01:38.141834 21007 raft_consensus.cc:1273] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Refusing update from remote peer 93b029e769a34e469c6bc80ad083f28b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:01:38.142014 21389 consensus_queue.cc:1035] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:38.142217 21355 consensus_queue.cc:1035] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:38.143713 21355 raft_consensus.cc:2953] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.144531 21084 raft_consensus.cc:2953] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.146440 20903 catalog_manager.cc:5582] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 reported cstate change: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "93b029e769a34e469c6bc80ad083f28b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.146591 21084 consensus_queue.cc:237] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.147928 21007 raft_consensus.cc:1273] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Refusing update from remote peer c089d4f3d49d4a24a66478348d576138: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 02:01:38.147953 21050 consensus_peers.cc:487] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 -> Peer 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132:45955): Couldn't send request to peer 8ee709b1b6bd409ba38922118e63f539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4517caf93a944de796fe4d6e5f7e22cb. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:01:38.148283 21159 raft_consensus.cc:1273] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Refusing update from remote peer c089d4f3d49d4a24a66478348d576138: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:01:38.148308 21241 consensus_queue.cc:1035] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:38.148730 21241 consensus_queue.cc:1035] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 02:01:38.149117 21125 consensus_peers.cc:487] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b -> Peer 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132:45955): Couldn't send request to peer 8ee709b1b6bd409ba38922118e63f539. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5564fd2ee2be4984823c0cc317d82a36. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:01:38.149272 21008 raft_consensus.cc:2953] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.150158 21381 raft_consensus.cc:2953] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.150619 21159 raft_consensus.cc:2953] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.152500 20903 catalog_manager.cc:5582] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 reported cstate change: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "c089d4f3d49d4a24a66478348d576138" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:38.153091 21008 raft_consensus.cc:2953] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee709b1b6bd409ba38922118e63f539 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } } }
I20250626 02:01:38.547178 21397 ts_tablet_manager.cc:927] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: Initiating tablet copy from peer 2afdb4ad450b40fbba9bf06deaf50715 (127.19.220.129:45863)
I20250626 02:01:38.547775 21397 tablet_copy_client.cc:323] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:45863
I20250626 02:01:38.551700 21018 tablet_copy_service.cc:140] P 2afdb4ad450b40fbba9bf06deaf50715: Received BeginTabletCopySession request for tablet fc511bed65b94bd7a4f024e0d56e92c9 from peer 8ee709b1b6bd409ba38922118e63f539 ({username='slave'} at 127.0.0.1:48976)
I20250626 02:01:38.551812 21018 tablet_copy_service.cc:161] P 2afdb4ad450b40fbba9bf06deaf50715: Beginning new tablet copy session on tablet fc511bed65b94bd7a4f024e0d56e92c9 from peer 8ee709b1b6bd409ba38922118e63f539 at {username='slave'} at 127.0.0.1:48976: session id = 8ee709b1b6bd409ba38922118e63f539-fc511bed65b94bd7a4f024e0d56e92c9
I20250626 02:01:38.552927 21018 tablet_copy_source_session.cc:215] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:01:38.554024 21397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc511bed65b94bd7a4f024e0d56e92c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:38.555903 21397 tablet_copy_client.cc:806] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Starting download of 0 data blocks...
I20250626 02:01:38.556084 21397 tablet_copy_client.cc:670] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Starting download of 1 WAL segments...
I20250626 02:01:38.557433 21397 tablet_copy_client.cc:538] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:01:38.563334 21397 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap starting.
I20250626 02:01:38.568616 21397 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:01:38.568892 21397 tablet_bootstrap.cc:492] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap complete.
I20250626 02:01:38.569077 21397 ts_tablet_manager.cc:1397] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20250626 02:01:38.569545 21397 raft_consensus.cc:357] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } attrs { replace: true } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.569659 21397 raft_consensus.cc:738] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee709b1b6bd409ba38922118e63f539, State: Initialized, Role: LEARNER
I20250626 02:01:38.569799 21397 consensus_queue.cc:260] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } attrs { replace: true } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.570195 21397 ts_tablet_manager.cc:1428] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:01:38.570801 21018 tablet_copy_service.cc:342] P 2afdb4ad450b40fbba9bf06deaf50715: Request end of tablet copy session 8ee709b1b6bd409ba38922118e63f539-fc511bed65b94bd7a4f024e0d56e92c9 received from {username='slave'} at 127.0.0.1:48976
I20250626 02:01:38.570901 21018 tablet_copy_service.cc:434] P 2afdb4ad450b40fbba9bf06deaf50715: ending tablet copy session 8ee709b1b6bd409ba38922118e63f539-fc511bed65b94bd7a4f024e0d56e92c9 on tablet fc511bed65b94bd7a4f024e0d56e92c9 with peer 8ee709b1b6bd409ba38922118e63f539
I20250626 02:01:38.581662 21397 ts_tablet_manager.cc:927] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: Initiating tablet copy from peer c089d4f3d49d4a24a66478348d576138 (127.19.220.130:46001)
I20250626 02:01:38.581991 21397 tablet_copy_client.cc:323] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:46001
I20250626 02:01:38.585602 21403 ts_tablet_manager.cc:927] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: Initiating tablet copy from peer 93b029e769a34e469c6bc80ad083f28b (127.19.220.131:45181)
I20250626 02:01:38.585606 21094 tablet_copy_service.cc:140] P c089d4f3d49d4a24a66478348d576138: Received BeginTabletCopySession request for tablet 1a2f4557c0074123a846d6b8b2e42a27 from peer 8ee709b1b6bd409ba38922118e63f539 ({username='slave'} at 127.0.0.1:36386)
I20250626 02:01:38.585768 21094 tablet_copy_service.cc:161] P c089d4f3d49d4a24a66478348d576138: Beginning new tablet copy session on tablet 1a2f4557c0074123a846d6b8b2e42a27 from peer 8ee709b1b6bd409ba38922118e63f539 at {username='slave'} at 127.0.0.1:36386: session id = 8ee709b1b6bd409ba38922118e63f539-1a2f4557c0074123a846d6b8b2e42a27
I20250626 02:01:38.585999 21403 tablet_copy_client.cc:323] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:45181
I20250626 02:01:38.586807 21094 tablet_copy_source_session.cc:215] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:01:38.587670 21397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2f4557c0074123a846d6b8b2e42a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:38.589303 21169 tablet_copy_service.cc:140] P 93b029e769a34e469c6bc80ad083f28b: Received BeginTabletCopySession request for tablet 5564fd2ee2be4984823c0cc317d82a36 from peer 8ee709b1b6bd409ba38922118e63f539 ({username='slave'} at 127.0.0.1:57828)
I20250626 02:01:38.589388 21169 tablet_copy_service.cc:161] P 93b029e769a34e469c6bc80ad083f28b: Beginning new tablet copy session on tablet 5564fd2ee2be4984823c0cc317d82a36 from peer 8ee709b1b6bd409ba38922118e63f539 at {username='slave'} at 127.0.0.1:57828: session id = 8ee709b1b6bd409ba38922118e63f539-5564fd2ee2be4984823c0cc317d82a36
I20250626 02:01:38.590045 21169 tablet_copy_source_session.cc:215] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:01:38.590159 21397 tablet_copy_client.cc:806] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Starting download of 0 data blocks...
I20250626 02:01:38.590353 21397 tablet_copy_client.cc:670] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Starting download of 1 WAL segments...
I20250626 02:01:38.590813 21403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5564fd2ee2be4984823c0cc317d82a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:38.591369 21397 tablet_copy_client.cc:538] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:01:38.600507 21397 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap starting.
I20250626 02:01:38.601194 21403 tablet_copy_client.cc:806] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Starting download of 0 data blocks...
I20250626 02:01:38.601409 21403 tablet_copy_client.cc:670] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Starting download of 1 WAL segments...
I20250626 02:01:38.602382 21403 tablet_copy_client.cc:538] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:01:38.604633 21397 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:01:38.604877 21397 tablet_bootstrap.cc:492] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap complete.
I20250626 02:01:38.605068 21397 ts_tablet_manager.cc:1397] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20250626 02:01:38.605523 21397 raft_consensus.cc:357] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } attrs { replace: true } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.605638 21397 raft_consensus.cc:738] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee709b1b6bd409ba38922118e63f539, State: Initialized, Role: LEARNER
I20250626 02:01:38.605751 21397 consensus_queue.cc:260] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } attrs { replace: true } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.606012 21397 ts_tablet_manager.cc:1428] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:38.606434 21094 tablet_copy_service.cc:342] P c089d4f3d49d4a24a66478348d576138: Request end of tablet copy session 8ee709b1b6bd409ba38922118e63f539-1a2f4557c0074123a846d6b8b2e42a27 received from {username='slave'} at 127.0.0.1:36386
I20250626 02:01:38.606513 21094 tablet_copy_service.cc:434] P c089d4f3d49d4a24a66478348d576138: ending tablet copy session 8ee709b1b6bd409ba38922118e63f539-1a2f4557c0074123a846d6b8b2e42a27 on tablet 1a2f4557c0074123a846d6b8b2e42a27 with peer 8ee709b1b6bd409ba38922118e63f539
I20250626 02:01:38.612617 21403 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap starting.
I20250626 02:01:38.617213 21403 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:01:38.617532 21403 tablet_bootstrap.cc:492] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: Bootstrap complete.
I20250626 02:01:38.617710 21403 ts_tablet_manager.cc:1397] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20250626 02:01:38.618132 21403 raft_consensus.cc:357] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.618265 21403 raft_consensus.cc:738] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee709b1b6bd409ba38922118e63f539, State: Initialized, Role: LEARNER
I20250626 02:01:38.618379 21403 consensus_queue.cc:260] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2afdb4ad450b40fbba9bf06deaf50715" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 45863 } attrs { replace: true } } peers { permanent_uuid: "c089d4f3d49d4a24a66478348d576138" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 46001 } } peers { permanent_uuid: "93b029e769a34e469c6bc80ad083f28b" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 45181 } } peers { permanent_uuid: "8ee709b1b6bd409ba38922118e63f539" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 45955 } attrs { promote: true } }
I20250626 02:01:38.618659 21403 ts_tablet_manager.cc:1428] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:38.619092 21169 tablet_copy_service.cc:342] P 93b029e769a34e469c6bc80ad083f28b: Request end of tablet copy session 8ee709b1b6bd409ba38922118e63f539-5564fd2ee2be4984823c0cc317d82a36 received from {username='slave'} at 127.0.0.1:57828
I20250626 02:01:38.619174 21169 tablet_copy_service.cc:434] P 93b029e769a34e469c6bc80ad083f28b: ending tablet copy session 8ee709b1b6bd409ba38922118e63f539-5564fd2ee2be4984823c0cc317d82a36 on tablet 5564fd2ee2be4984823c0cc317d82a36 with peer 8ee709b1b6bd409ba38922118e63f539
I20250626 02:01:38.662334 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:01:38.666297 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:38.666517 20338 tablet_replica.cc:331] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715: stopping tablet replica
I20250626 02:01:38.666669 20338 raft_consensus.cc:2241] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.666807 20338 raft_consensus.cc:2270] T 1a2f4557c0074123a846d6b8b2e42a27 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 02:01:38.667188 20892 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20250626 02:01:38.667390 20338 tablet_replica.cc:331] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715: stopping tablet replica
I20250626 02:01:38.667513 20338 raft_consensus.cc:2241] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.667616 20338 raft_consensus.cc:2270] T fa76cd3c1c3c46ab981b3ba36b57b940 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.667974 20338 tablet_replica.cc:331] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715: stopping tablet replica
W20250626 02:01:38.667999 20953 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:01:38.668056 20953 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250626 02:01:38.668062 20338 raft_consensus.cc:2241] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.668213 20338 raft_consensus.cc:2270] T 5564fd2ee2be4984823c0cc317d82a36 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.668669 20338 tablet_replica.cc:331] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715: stopping tablet replica
I20250626 02:01:38.668771 20338 raft_consensus.cc:2241] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.668936 20338 raft_consensus.cc:2270] T 12d287a85264480783ca51bf9cda9218 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.669274 20338 tablet_replica.cc:331] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715: stopping tablet replica
I20250626 02:01:38.669366 20338 raft_consensus.cc:2241] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.669474 20338 raft_consensus.cc:2270] T 4517caf93a944de796fe4d6e5f7e22cb P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.669951 20338 tablet_replica.cc:331] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715: stopping tablet replica
I20250626 02:01:38.670042 20338 raft_consensus.cc:2241] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.670117 20338 raft_consensus.cc:2270] T f131b26b4fb844faab3e91abf00f6a65 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.670431 20338 tablet_replica.cc:331] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715: stopping tablet replica
I20250626 02:01:38.670500 20338 raft_consensus.cc:2241] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.670723 20338 raft_consensus.cc:2270] T fc511bed65b94bd7a4f024e0d56e92c9 P 2afdb4ad450b40fbba9bf06deaf50715 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.671139 20338 tablet_replica.cc:331] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715: stopping tablet replica
I20250626 02:01:38.671226 20338 raft_consensus.cc:2241] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.671382 20338 raft_consensus.cc:2270] T be32c1f8be4f4267999798b3a7f83814 P 2afdb4ad450b40fbba9bf06deaf50715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.685170 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:01:38.688117 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
W20250626 02:01:38.690065 20953 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:01:38.690135 20953 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:01:38.691219 20953 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:01:38.691275 20953 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250626 02:01:38.693892 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:38.694049 20338 tablet_replica.cc:331] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138: stopping tablet replica
I20250626 02:01:38.694190 20338 raft_consensus.cc:2241] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.694396 20338 raft_consensus.cc:2270] T 4517caf93a944de796fe4d6e5f7e22cb P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.695117 20338 tablet_replica.cc:331] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138: stopping tablet replica
I20250626 02:01:38.695242 20338 raft_consensus.cc:2241] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.695344 20338 raft_consensus.cc:2270] T 12d287a85264480783ca51bf9cda9218 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.695705 20338 tablet_replica.cc:331] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138: stopping tablet replica
I20250626 02:01:38.695803 20338 raft_consensus.cc:2241] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.695996 20338 raft_consensus.cc:2270] T 1a2f4557c0074123a846d6b8b2e42a27 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.696521 20338 tablet_replica.cc:331] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138: stopping tablet replica
I20250626 02:01:38.696616 20338 raft_consensus.cc:2241] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.696715 20338 raft_consensus.cc:2270] T 5564fd2ee2be4984823c0cc317d82a36 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.697176 20338 tablet_replica.cc:331] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138: stopping tablet replica
I20250626 02:01:38.697259 20338 raft_consensus.cc:2241] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.697352 20338 raft_consensus.cc:2270] T be32c1f8be4f4267999798b3a7f83814 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.697676 20338 tablet_replica.cc:331] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138: stopping tablet replica
I20250626 02:01:38.697774 20338 raft_consensus.cc:2241] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.697914 20338 raft_consensus.cc:2270] T fa76cd3c1c3c46ab981b3ba36b57b940 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.698215 20338 tablet_replica.cc:331] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138: stopping tablet replica
I20250626 02:01:38.698313 20338 raft_consensus.cc:2241] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.698406 20338 raft_consensus.cc:2270] T f131b26b4fb844faab3e91abf00f6a65 P c089d4f3d49d4a24a66478348d576138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.698725 20338 tablet_replica.cc:331] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138: stopping tablet replica
I20250626 02:01:38.698793 20338 raft_consensus.cc:2241] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.698889 20338 raft_consensus.cc:2270] T fc511bed65b94bd7a4f024e0d56e92c9 P c089d4f3d49d4a24a66478348d576138 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.712605 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:01:38.715062 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
W20250626 02:01:38.716750 20953 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:01:38.716904 20953 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250626 02:01:38.719558 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:38.719699 20338 tablet_replica.cc:331] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b: stopping tablet replica
I20250626 02:01:38.719820 20338 raft_consensus.cc:2241] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.720016 20338 raft_consensus.cc:2270] T 5564fd2ee2be4984823c0cc317d82a36 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.720528 20338 tablet_replica.cc:331] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b: stopping tablet replica
I20250626 02:01:38.720636 20338 raft_consensus.cc:2241] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.720731 20338 raft_consensus.cc:2270] T be32c1f8be4f4267999798b3a7f83814 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.721043 20338 tablet_replica.cc:331] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b: stopping tablet replica
I20250626 02:01:38.721257 20338 raft_consensus.cc:2241] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.721359 20338 raft_consensus.cc:2270] T 12d287a85264480783ca51bf9cda9218 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.721693 20338 tablet_replica.cc:331] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b: stopping tablet replica
I20250626 02:01:38.721796 20338 raft_consensus.cc:2241] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.721895 20338 raft_consensus.cc:2270] T 1a2f4557c0074123a846d6b8b2e42a27 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.722280 20338 tablet_replica.cc:331] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b: stopping tablet replica
I20250626 02:01:38.722374 20338 raft_consensus.cc:2241] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.722460 20338 raft_consensus.cc:2270] T fa76cd3c1c3c46ab981b3ba36b57b940 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.722733 20338 tablet_replica.cc:331] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b: stopping tablet replica
I20250626 02:01:38.722836 20338 raft_consensus.cc:2241] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.722926 20338 raft_consensus.cc:2270] T fc511bed65b94bd7a4f024e0d56e92c9 P 93b029e769a34e469c6bc80ad083f28b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.723297 20338 tablet_replica.cc:331] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b: stopping tablet replica
I20250626 02:01:38.723384 20338 raft_consensus.cc:2241] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.723547 20338 raft_consensus.cc:2270] T f131b26b4fb844faab3e91abf00f6a65 P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.723826 20338 tablet_replica.cc:331] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b: stopping tablet replica
I20250626 02:01:38.723915 20338 raft_consensus.cc:2241] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:38.724009 20338 raft_consensus.cc:2270] T 4517caf93a944de796fe4d6e5f7e22cb P 93b029e769a34e469c6bc80ad083f28b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.737452 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:01:38.739883 20338 tablet_server.cc:178] TabletServer@127.19.220.132:0 shutting down...
I20250626 02:01:38.743702 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:38.743933 20338 tablet_replica.cc:331] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539: stopping tablet replica
I20250626 02:01:38.744084 20338 raft_consensus.cc:2241] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539 [term 2 LEARNER]: Raft consensus shutting down.
I20250626 02:01:38.744235 20338 raft_consensus.cc:2270] T fc511bed65b94bd7a4f024e0d56e92c9 P 8ee709b1b6bd409ba38922118e63f539 [term 2 LEARNER]: Raft consensus is shut down!
I20250626 02:01:38.744725 20338 tablet_replica.cc:331] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539: stopping tablet replica
I20250626 02:01:38.744825 20338 raft_consensus.cc:2241] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539 [term 1 LEARNER]: Raft consensus shutting down.
I20250626 02:01:38.744920 20338 raft_consensus.cc:2270] T 5564fd2ee2be4984823c0cc317d82a36 P 8ee709b1b6bd409ba38922118e63f539 [term 1 LEARNER]: Raft consensus is shut down!
I20250626 02:01:38.745265 20338 tablet_replica.cc:331] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539: stopping tablet replica
I20250626 02:01:38.745358 20338 raft_consensus.cc:2241] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539 [term 1 LEARNER]: Raft consensus shutting down.
I20250626 02:01:38.745442 20338 raft_consensus.cc:2270] T 1a2f4557c0074123a846d6b8b2e42a27 P 8ee709b1b6bd409ba38922118e63f539 [term 1 LEARNER]: Raft consensus is shut down!
I20250626 02:01:38.757863 20338 tablet_server.cc:195] TabletServer@127.19.220.132:0 shutdown complete.
I20250626 02:01:38.759740 20338 master.cc:561] Master@127.19.220.190:34233 shutting down...
I20250626 02:01:38.763411 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:38.763547 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:38.763597 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P d1da6d3852a3414baa28859e57edf1fd: stopping tablet replica
I20250626 02:01:38.776315 20338 master.cc:583] Master@127.19.220.190:34233 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7720 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20250626 02:01:38.787331 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:38219,127.19.220.189:37117,127.19.220.188:46299
I20250626 02:01:38.787712 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:38.789196 21409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:38.789286 21411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:38.789444 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:38.789229 21408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:38.789650 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:38.789698 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:38.789713 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903298789714 us; error 0 us; skew 500 ppm
I20250626 02:01:38.790593 20338 webserver.cc:469] Webserver started at http://127.19.220.190:38789/ using document root <none> and password file <none>
I20250626 02:01:38.790771 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:38.790817 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:38.790899 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:38.791275 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.792713 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:01:38.793535 21416 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.793771 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:38.793830 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.793900 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:38.815222 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:38.815639 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:38.819757 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:38219
I20250626 02:01:38.820983 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:01:38.825464 21468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:38219 every 8 connection(s)
I20250626 02:01:38.825976 21469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 02:01:38.826467 21471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:38.826514 21472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:38.826699 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:38.826785 21474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:38.827032 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:38.827075 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:38.827091 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903298827091 us; error 0 us; skew 500 ppm
I20250626 02:01:38.827512 21469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:38.828248 20338 webserver.cc:469] Webserver started at http://127.19.220.189:33141/ using document root <none> and password file <none>
I20250626 02:01:38.828403 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:38.828462 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:38.828548 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:38.828953 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root/instance:
uuid: "90b1dbff36f9442090e578971f08e6b5"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.830682 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:01:38.831743 21481 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.831987 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:38.832044 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root
uuid: "90b1dbff36f9442090e578971f08e6b5"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.832109 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:38.832530 21469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } has no permanent_uuid. Determining permanent_uuid...
W20250626 02:01:38.833885 21469 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.220.189:37117: Network error: Client connection negotiation failed: client connection to 127.19.220.189:37117: connect: Connection refused (error 111)
I20250626 02:01:38.839015 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:38.839391 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:38.843124 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.189:37117
I20250626 02:01:38.843147 21534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.189:37117 every 8 connection(s)
I20250626 02:01:38.843847 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:01:38.843992 21535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:38.845432 21535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } has no permanent_uuid. Determining permanent_uuid...
W20250626 02:01:38.845557 21537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:38.845579 21540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:38.845602 21538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:38.846155 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:38.846314 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:38.846361 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:38.846381 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903298846381 us; error 0 us; skew 500 ppm
I20250626 02:01:38.847412 20338 webserver.cc:469] Webserver started at http://127.19.220.188:46433/ using document root <none> and password file <none>
I20250626 02:01:38.847570 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:38.847621 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:38.847692 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:38.848058 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root/instance:
uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.849836 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20250626 02:01:38.849853 21535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:38.851078 21547 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.851336 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:38.851400 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root
uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.851464 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:38.853981 21535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } has no permanent_uuid. Determining permanent_uuid...
W20250626 02:01:38.855185 21535 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.220.188:46299: Network error: Client connection negotiation failed: client connection to 127.19.220.188:46299: connect: Connection refused (error 111)
I20250626 02:01:38.861222 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:38.861589 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:38.864032 21469 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } attempt: 1
I20250626 02:01:38.865859 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.188:46299
I20250626 02:01:38.865869 21600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.188:46299 every 8 connection(s)
I20250626 02:01:38.866372 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:01:38.866786 21601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:38.868247 21601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:38.868450 21469 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:38.872099 21601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:38.873337 21469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: Bootstrap starting.
I20250626 02:01:38.874336 21469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:38.875562 21469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: No bootstrap required, opened a new log
I20250626 02:01:38.875685 21601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } has no permanent_uuid. Determining permanent_uuid...
I20250626 02:01:38.876036 21469 raft_consensus.cc:357] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.876169 21469 raft_consensus.cc:383] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:38.876209 21469 raft_consensus.cc:738] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f3dd3fcdf4491ab844a80de8e30a3a, State: Initialized, Role: FOLLOWER
I20250626 02:01:38.876363 21469 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.876662 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.876776 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:38.876804 21469 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:01:38.878590 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250626 02:01:38.879163 21617 catalog_manager.cc:1560] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 02:01:38.879240 21617 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 02:01:38.879946 21601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8: Bootstrap starting.
I20250626 02:01:38.880805 21601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:38.881747 21601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8: No bootstrap required, opened a new log
I20250626 02:01:38.882090 21601 raft_consensus.cc:357] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.882169 21601 raft_consensus.cc:383] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:38.882197 21601 raft_consensus.cc:738] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c7ed0f056e44aeb3e500a5062fbbb8, State: Initialized, Role: FOLLOWER
I20250626 02:01:38.882283 21601 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.882529 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.882639 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:38.882638 21601 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: configured and running, proceeding with master startup.
W20250626 02:01:38.884617 21630 catalog_manager.cc:1560] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 02:01:38.884681 21630 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 02:01:38.893514 21619 raft_consensus.cc:491] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:38.893638 21619 raft_consensus.cc:513] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.894085 21619 leader_election.cc:290] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5f3dd3fcdf4491ab844a80de8e30a3a (127.19.220.190:38219), 90b1dbff36f9442090e578971f08e6b5 (127.19.220.189:37117)
I20250626 02:01:38.894519 21443 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" is_pre_election: true
I20250626 02:01:38.894675 21443 raft_consensus.cc:2466] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3c7ed0f056e44aeb3e500a5062fbbb8 in term 0.
I20250626 02:01:38.894973 21551 leader_election.cc:304] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f3dd3fcdf4491ab844a80de8e30a3a, c3c7ed0f056e44aeb3e500a5062fbbb8; no voters:
I20250626 02:01:38.895135 21619 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:38.895190 21619 raft_consensus.cc:491] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:38.895228 21619 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:38.896071 21619 raft_consensus.cc:513] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.896436 21619 leader_election.cc:290] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [CANDIDATE]: Term 1 election: Requested vote from peers a5f3dd3fcdf4491ab844a80de8e30a3a (127.19.220.190:38219), 90b1dbff36f9442090e578971f08e6b5 (127.19.220.189:37117)
I20250626 02:01:38.896736 21443 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a"
I20250626 02:01:38.896848 21443 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:38.897848 21443 raft_consensus.cc:2466] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3c7ed0f056e44aeb3e500a5062fbbb8 in term 1.
I20250626 02:01:38.898138 21551 leader_election.cc:304] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f3dd3fcdf4491ab844a80de8e30a3a, c3c7ed0f056e44aeb3e500a5062fbbb8; no voters:
I20250626 02:01:38.898301 21619 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:38.898530 21619 raft_consensus.cc:695] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 1 LEADER]: Becoming Leader. State: Replica: c3c7ed0f056e44aeb3e500a5062fbbb8, State: Running, Role: LEADER
I20250626 02:01:38.898674 21619 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.899381 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c7ed0f056e44aeb3e500a5062fbbb8. Latest consensus state: current_term: 1 leader_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.899518 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:38.899863 21633 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:01:38.900763 21633 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:01:38.902711 21443 raft_consensus.cc:1273] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 1 FOLLOWER]: Refusing update from remote peer c3c7ed0f056e44aeb3e500a5062fbbb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:01:38.903092 21631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:38.904456 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c7ed0f056e44aeb3e500a5062fbbb8. Latest consensus state: current_term: 1 leader_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.904567 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:38.905519 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.905619 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:38.905823 21633 catalog_manager.cc:1349] Generated new cluster ID: f5ad075837e0432182f32bed7dedf44b
I20250626 02:01:38.905889 21633 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:01:38.906358 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.906452 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:38.919348 21535 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } attempt: 1
I20250626 02:01:38.925472 21535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5: Bootstrap starting.
I20250626 02:01:38.926693 21535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:38.928105 21535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5: No bootstrap required, opened a new log
I20250626 02:01:38.928653 21535 raft_consensus.cc:357] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.928768 21535 raft_consensus.cc:383] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:38.928817 21535 raft_consensus.cc:738] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b1dbff36f9442090e578971f08e6b5, State: Initialized, Role: FOLLOWER
I20250626 02:01:38.928954 21535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:38.929337 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.929508 21535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:01:38.929600 21509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b1dbff36f9442090e578971f08e6b5" is_pre_election: true
I20250626 02:01:38.929666 21507 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:38.929589 21508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b1dbff36f9442090e578971f08e6b5"
I20250626 02:01:38.930830 21507 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 1 FOLLOWER]: Refusing update from remote peer c3c7ed0f056e44aeb3e500a5062fbbb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:01:38.931275 21619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:38.931488 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250626 02:01:38.932024 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:38.932046 21651 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 02:01:38.932096 21651 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 02:01:38.933279 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:38.933250 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c7ed0f056e44aeb3e500a5062fbbb8. Latest consensus state: current_term: 1 leader_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.933367 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 02:01:38.933406 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.933511 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [sys.catalog]: This master's current role is: FOLLOWER
W20250626 02:01:38.933980 21655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:38.935184 21657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:38.935472 21660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:38.935508 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:38.935679 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:38.935732 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:38.935751 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903298935751 us; error 0 us; skew 500 ppm
I20250626 02:01:38.936741 20338 webserver.cc:469] Webserver started at http://127.19.220.129:43065/ using document root <none> and password file <none>
I20250626 02:01:38.936880 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:38.936928 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:38.936990 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:38.937410 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "3003bea79883450f819ac70c12cf3835"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.938942 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20250626 02:01:38.939867 21665 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.940140 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:38.940248 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "3003bea79883450f819ac70c12cf3835"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.940308 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:38.950945 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:38.951438 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:38.951778 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:38.952380 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:38.952423 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.952468 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:38.952493 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.953516 21673 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:38.953639 21673 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:38.956029 21633 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:01:38.956645 21633 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:01:38.958485 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:42083
I20250626 02:01:38.958537 21729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:42083 every 8 connection(s)
I20250626 02:01:38.959489 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:38.961963 21735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:38.962050 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:38.962158 21738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:38.962755 21736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:38.963294 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:38.963393 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:38.963465 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903298963464 us; error 0 us; skew 500 ppm
I20250626 02:01:38.964972 20338 webserver.cc:469] Webserver started at http://127.19.220.130:42441/ using document root <none> and password file <none>
I20250626 02:01:38.965139 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:38.965214 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:38.965286 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:38.965401 21730 heartbeater.cc:344] Connected to a master server at 127.19.220.188:46299
I20250626 02:01:38.965492 21730 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:38.965727 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "d5673596f3784121a7c55bb45c46645c"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.965709 21730 heartbeater.cc:507] Master 127.19.220.188:46299 requested a full tablet report, sending...
I20250626 02:01:38.966141 21633 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8: Generated new TSK 0
I20250626 02:01:38.966346 21633 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:01:38.966813 21565 ts_manager.cc:194] Registered new tserver with Master: 3003bea79883450f819ac70c12cf3835 (127.19.220.129:42083)
I20250626 02:01:38.967670 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250626 02:01:38.968917 21745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.969182 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:38.969264 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "d5673596f3784121a7c55bb45c46645c"
format_stamp: "Formatted at 2025-06-26 02:01:38 on dist-test-slave-h18f"
I20250626 02:01:38.969328 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:38.969765 21731 heartbeater.cc:344] Connected to a master server at 127.19.220.190:38219
I20250626 02:01:38.969835 21731 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:38.970069 21731 heartbeater.cc:507] Master 127.19.220.190:38219 requested a full tablet report, sending...
I20250626 02:01:38.970669 21433 ts_manager.cc:194] Registered new tserver with Master: 3003bea79883450f819ac70c12cf3835 (127.19.220.129:42083)
I20250626 02:01:38.972684 21732 heartbeater.cc:344] Connected to a master server at 127.19.220.189:37117
I20250626 02:01:38.972759 21732 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:38.972944 21732 heartbeater.cc:507] Master 127.19.220.189:37117 requested a full tablet report, sending...
I20250626 02:01:38.973537 21499 ts_manager.cc:194] Registered new tserver with Master: 3003bea79883450f819ac70c12cf3835 (127.19.220.129:42083)
I20250626 02:01:38.993778 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:38.994227 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:38.994542 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:38.995049 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:38.995092 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.995146 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:38.995182 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:38.999745 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:33447
I20250626 02:01:38.999792 21808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:33447 every 8 connection(s)
I20250626 02:01:39.000761 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:39.003135 21816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:39.003410 21819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:39.003998 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:39.004118 21817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:39.004618 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:39.004673 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:39.004693 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903299004693 us; error 0 us; skew 500 ppm
I20250626 02:01:39.005324 21810 heartbeater.cc:344] Connected to a master server at 127.19.220.190:38219
I20250626 02:01:39.005437 21810 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:39.005697 21810 heartbeater.cc:507] Master 127.19.220.190:38219 requested a full tablet report, sending...
I20250626 02:01:39.005796 20338 webserver.cc:469] Webserver started at http://127.19.220.131:35371/ using document root <none> and password file <none>
I20250626 02:01:39.005975 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:39.006038 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:39.006114 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:39.006314 21811 heartbeater.cc:344] Connected to a master server at 127.19.220.189:37117
I20250626 02:01:39.006436 21811 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:39.006443 21433 ts_manager.cc:194] Registered new tserver with Master: d5673596f3784121a7c55bb45c46645c (127.19.220.130:33447)
I20250626 02:01:39.006572 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "9fb255bf645c4c51b195f3d35597e72a"
format_stamp: "Formatted at 2025-06-26 02:01:39 on dist-test-slave-h18f"
I20250626 02:01:39.006734 21809 heartbeater.cc:344] Connected to a master server at 127.19.220.188:46299
I20250626 02:01:39.006814 21809 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:39.006974 21809 heartbeater.cc:507] Master 127.19.220.188:46299 requested a full tablet report, sending...
I20250626 02:01:39.006975 21811 heartbeater.cc:507] Master 127.19.220.189:37117 requested a full tablet report, sending...
I20250626 02:01:39.007527 21565 ts_manager.cc:194] Registered new tserver with Master: d5673596f3784121a7c55bb45c46645c (127.19.220.130:33447)
I20250626 02:01:39.007709 21499 ts_manager.cc:194] Registered new tserver with Master: d5673596f3784121a7c55bb45c46645c (127.19.220.130:33447)
I20250626 02:01:39.008392 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20250626 02:01:39.008443 21565 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42690
I20250626 02:01:39.009346 21824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:39.009604 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:39.009670 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "9fb255bf645c4c51b195f3d35597e72a"
format_stamp: "Formatted at 2025-06-26 02:01:39 on dist-test-slave-h18f"
I20250626 02:01:39.009734 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:39.023350 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:39.023798 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:39.024089 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:39.024582 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:39.024624 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:39.024677 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:39.024709 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:39.029055 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:36995
I20250626 02:01:39.029175 21887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:36995 every 8 connection(s)
I20250626 02:01:39.034205 21889 heartbeater.cc:344] Connected to a master server at 127.19.220.190:38219
I20250626 02:01:39.034286 21890 heartbeater.cc:344] Connected to a master server at 127.19.220.189:37117
I20250626 02:01:39.034312 21889 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:39.034341 21890 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:39.034421 21888 heartbeater.cc:344] Connected to a master server at 127.19.220.188:46299
I20250626 02:01:39.034494 21888 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:39.034543 21889 heartbeater.cc:507] Master 127.19.220.190:38219 requested a full tablet report, sending...
I20250626 02:01:39.034543 21890 heartbeater.cc:507] Master 127.19.220.189:37117 requested a full tablet report, sending...
I20250626 02:01:39.034639 21888 heartbeater.cc:507] Master 127.19.220.188:46299 requested a full tablet report, sending...
I20250626 02:01:39.035189 21433 ts_manager.cc:194] Registered new tserver with Master: 9fb255bf645c4c51b195f3d35597e72a (127.19.220.131:36995)
I20250626 02:01:39.035197 21565 ts_manager.cc:194] Registered new tserver with Master: 9fb255bf645c4c51b195f3d35597e72a (127.19.220.131:36995)
I20250626 02:01:39.035189 21499 ts_manager.cc:194] Registered new tserver with Master: 9fb255bf645c4c51b195f3d35597e72a (127.19.220.131:36995)
I20250626 02:01:39.035354 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005738997s
I20250626 02:01:39.035908 21565 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42698
I20250626 02:01:39.879884 21617 catalog_manager.cc:1261] Loaded cluster ID: f5ad075837e0432182f32bed7dedf44b
I20250626 02:01:39.879963 21617 catalog_manager.cc:1554] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: loading cluster ID for follower catalog manager: success
I20250626 02:01:39.880995 21617 catalog_manager.cc:1576] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: acquiring CA information for follower catalog manager: success
I20250626 02:01:39.881516 21617 catalog_manager.cc:1604] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 02:01:39.932941 21651 catalog_manager.cc:1261] Loaded cluster ID: f5ad075837e0432182f32bed7dedf44b
I20250626 02:01:39.933015 21651 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5: loading cluster ID for follower catalog manager: success
I20250626 02:01:39.934270 21651 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5: acquiring CA information for follower catalog manager: success
I20250626 02:01:39.934815 21651 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 02:01:39.968734 21565 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42686
I20250626 02:01:40.009159 21809 heartbeater.cc:499] Master 127.19.220.188:46299 was elected leader, sending a full tablet report...
I20250626 02:01:40.036716 21888 heartbeater.cc:499] Master 127.19.220.188:46299 was elected leader, sending a full tablet report...
I20250626 02:01:40.884334 21629 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:40.969687 21730 heartbeater.cc:499] Master 127.19.220.188:46299 was elected leader, sending a full tablet report...
I20250626 02:01:41.090121 20338 test_util.cc:276] Using random seed: -246666427
I20250626 02:01:41.099311 21565 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42706:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 02:01:41.114005 21773 tablet_service.cc:1468] Processing CreateTablet for tablet afcfb59ea16b43cbaed8bb2d61158bc4 (DEFAULT_TABLE table=test-workload [id=db68d16267da4c9a919c7ab9fe1d50ea]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250626 02:01:41.114128 21694 tablet_service.cc:1468] Processing CreateTablet for tablet 668623f6f3f74deda264481fe2fb07f7 (DEFAULT_TABLE table=test-workload [id=db68d16267da4c9a919c7ab9fe1d50ea]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250626 02:01:41.114343 21773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afcfb59ea16b43cbaed8bb2d61158bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:41.114343 21694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 668623f6f3f74deda264481fe2fb07f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:41.114511 21852 tablet_service.cc:1468] Processing CreateTablet for tablet 6c6147cb54c343bf98ae4c28786e06fe (DEFAULT_TABLE table=test-workload [id=db68d16267da4c9a919c7ab9fe1d50ea]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250626 02:01:41.114790 21852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c6147cb54c343bf98ae4c28786e06fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:41.116636 21925 tablet_bootstrap.cc:492] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835: Bootstrap starting.
I20250626 02:01:41.116731 21927 tablet_bootstrap.cc:492] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a: Bootstrap starting.
I20250626 02:01:41.117478 21925 tablet_bootstrap.cc:654] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:41.117509 21927 tablet_bootstrap.cc:654] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:41.118027 21926 tablet_bootstrap.cc:492] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c: Bootstrap starting.
I20250626 02:01:41.118440 21925 tablet_bootstrap.cc:492] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835: No bootstrap required, opened a new log
I20250626 02:01:41.118440 21927 tablet_bootstrap.cc:492] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a: No bootstrap required, opened a new log
I20250626 02:01:41.118571 21927 ts_tablet_manager.cc:1397] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:41.118571 21925 ts_tablet_manager.cc:1397] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:01:41.118824 21926 tablet_bootstrap.cc:654] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:41.119041 21925 raft_consensus.cc:357] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3003bea79883450f819ac70c12cf3835" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 42083 } }
I20250626 02:01:41.119004 21927 raft_consensus.cc:357] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb255bf645c4c51b195f3d35597e72a" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36995 } }
I20250626 02:01:41.119127 21927 raft_consensus.cc:383] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:41.119117 21925 raft_consensus.cc:383] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:41.119277 21927 raft_consensus.cc:738] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb255bf645c4c51b195f3d35597e72a, State: Initialized, Role: FOLLOWER
I20250626 02:01:41.119290 21925 raft_consensus.cc:738] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3003bea79883450f819ac70c12cf3835, State: Initialized, Role: FOLLOWER
I20250626 02:01:41.119486 21927 consensus_queue.cc:260] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb255bf645c4c51b195f3d35597e72a" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36995 } }
I20250626 02:01:41.119468 21925 consensus_queue.cc:260] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3003bea79883450f819ac70c12cf3835" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 42083 } }
I20250626 02:01:41.119575 21927 raft_consensus.cc:397] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:41.119582 21925 raft_consensus.cc:397] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:41.119632 21927 raft_consensus.cc:491] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:41.119637 21925 raft_consensus.cc:491] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:41.119695 21925 raft_consensus.cc:3058] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:41.119694 21927 raft_consensus.cc:3058] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:41.120590 21926 tablet_bootstrap.cc:492] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c: No bootstrap required, opened a new log
I20250626 02:01:41.120675 21926 ts_tablet_manager.cc:1397] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250626 02:01:41.120659 21927 raft_consensus.cc:513] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb255bf645c4c51b195f3d35597e72a" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36995 } }
I20250626 02:01:41.120659 21925 raft_consensus.cc:513] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3003bea79883450f819ac70c12cf3835" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 42083 } }
I20250626 02:01:41.120857 21927 leader_election.cc:304] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fb255bf645c4c51b195f3d35597e72a; no voters:
I20250626 02:01:41.120857 21925 leader_election.cc:304] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3003bea79883450f819ac70c12cf3835; no voters:
I20250626 02:01:41.121084 21925 leader_election.cc:290] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:41.121084 21927 leader_election.cc:290] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:41.121114 21926 raft_consensus.cc:357] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5673596f3784121a7c55bb45c46645c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33447 } }
I20250626 02:01:41.121255 21926 raft_consensus.cc:383] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:41.121302 21926 raft_consensus.cc:738] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5673596f3784121a7c55bb45c46645c, State: Initialized, Role: FOLLOWER
I20250626 02:01:41.121450 21927 ts_tablet_manager.cc:1428] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250626 02:01:41.121493 21931 raft_consensus.cc:2802] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:41.121446 21925 ts_tablet_manager.cc:1428] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250626 02:01:41.121435 21932 raft_consensus.cc:2802] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:41.121707 21931 raft_consensus.cc:695] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 1 LEADER]: Becoming Leader. State: Replica: 9fb255bf645c4c51b195f3d35597e72a, State: Running, Role: LEADER
I20250626 02:01:41.121757 21932 raft_consensus.cc:695] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 1 LEADER]: Becoming Leader. State: Replica: 3003bea79883450f819ac70c12cf3835, State: Running, Role: LEADER
I20250626 02:01:41.121821 21931 consensus_queue.cc:237] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb255bf645c4c51b195f3d35597e72a" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36995 } }
I20250626 02:01:41.121865 21932 consensus_queue.cc:237] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3003bea79883450f819ac70c12cf3835" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 42083 } }
I20250626 02:01:41.121495 21926 consensus_queue.cc:260] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5673596f3784121a7c55bb45c46645c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33447 } }
I20250626 02:01:41.122121 21926 raft_consensus.cc:397] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:41.122175 21926 raft_consensus.cc:491] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:41.122231 21926 raft_consensus.cc:3058] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:41.123440 21565 catalog_manager.cc:5582] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3003bea79883450f819ac70c12cf3835 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "3003bea79883450f819ac70c12cf3835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3003bea79883450f819ac70c12cf3835" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 42083 } health_report { overall_health: HEALTHY } } }
I20250626 02:01:41.124114 21564 catalog_manager.cc:5582] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fb255bf645c4c51b195f3d35597e72a (127.19.220.131). New cstate: current_term: 1 leader_uuid: "9fb255bf645c4c51b195f3d35597e72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb255bf645c4c51b195f3d35597e72a" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 36995 } health_report { overall_health: HEALTHY } } }
I20250626 02:01:41.126274 21926 raft_consensus.cc:513] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5673596f3784121a7c55bb45c46645c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33447 } }
I20250626 02:01:41.126390 21926 leader_election.cc:304] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5673596f3784121a7c55bb45c46645c; no voters:
I20250626 02:01:41.126613 21926 leader_election.cc:290] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:41.126744 21939 raft_consensus.cc:2802] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:41.126953 21939 raft_consensus.cc:695] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 1 LEADER]: Becoming Leader. State: Replica: d5673596f3784121a7c55bb45c46645c, State: Running, Role: LEADER
I20250626 02:01:41.127108 21939 consensus_queue.cc:237] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5673596f3784121a7c55bb45c46645c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33447 } }
I20250626 02:01:41.127568 21926 ts_tablet_manager.cc:1428] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250626 02:01:41.130123 21565 catalog_manager.cc:5582] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c reported cstate change: term changed from 0 to 1, leader changed from <none> to d5673596f3784121a7c55bb45c46645c (127.19.220.130). New cstate: current_term: 1 leader_uuid: "d5673596f3784121a7c55bb45c46645c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5673596f3784121a7c55bb45c46645c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33447 } health_report { overall_health: HEALTHY } } }
I20250626 02:01:41.137274 20338 master.cc:561] Master@127.19.220.188:46299 shutting down...
I20250626 02:01:41.139847 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:41.140010 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:41.140085 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P c3c7ed0f056e44aeb3e500a5062fbbb8: stopping tablet replica
I20250626 02:01:41.153424 20338 master.cc:583] Master@127.19.220.188:46299 shutdown complete.
W20250626 02:01:42.133553 21730 heartbeater.cc:646] Failed to heartbeat to 127.19.220.188:46299 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.220.188:46299: connect: Connection refused (error 111)
I20250626 02:01:42.635038 21949 raft_consensus.cc:491] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3c7ed0f056e44aeb3e500a5062fbbb8)
I20250626 02:01:42.635193 21949 raft_consensus.cc:513] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:42.635620 21949 leader_election.cc:290] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90b1dbff36f9442090e578971f08e6b5 (127.19.220.189:37117), c3c7ed0f056e44aeb3e500a5062fbbb8 (127.19.220.188:46299)
I20250626 02:01:42.635974 21950 raft_consensus.cc:491] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3c7ed0f056e44aeb3e500a5062fbbb8)
I20250626 02:01:42.636078 21950 raft_consensus.cc:513] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:42.636189 21507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "90b1dbff36f9442090e578971f08e6b5" is_pre_election: true
I20250626 02:01:42.636286 21507 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f3dd3fcdf4491ab844a80de8e30a3a in term 1.
I20250626 02:01:42.636574 21419 leader_election.cc:304] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b1dbff36f9442090e578971f08e6b5, a5f3dd3fcdf4491ab844a80de8e30a3a; no voters:
I20250626 02:01:42.636631 21950 leader_election.cc:290] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5f3dd3fcdf4491ab844a80de8e30a3a (127.19.220.190:38219), c3c7ed0f056e44aeb3e500a5062fbbb8 (127.19.220.188:46299)
I20250626 02:01:42.636736 21949 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 02:01:42.636798 21949 raft_consensus.cc:491] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader c3c7ed0f056e44aeb3e500a5062fbbb8)
I20250626 02:01:42.636842 21949 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:42.636739 21443 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90b1dbff36f9442090e578971f08e6b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" is_pre_election: true
W20250626 02:01:42.637215 21418 leader_election.cc:336] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c3c7ed0f056e44aeb3e500a5062fbbb8 (127.19.220.188:46299): Network error: Client connection negotiation failed: client connection to 127.19.220.188:46299: connect: Connection refused (error 111)
W20250626 02:01:42.637835 21483 leader_election.cc:336] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c3c7ed0f056e44aeb3e500a5062fbbb8 (127.19.220.188:46299): Network error: Client connection negotiation failed: client connection to 127.19.220.188:46299: connect: Connection refused (error 111)
I20250626 02:01:42.639771 21949 raft_consensus.cc:513] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:42.639983 21443 raft_consensus.cc:2391] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90b1dbff36f9442090e578971f08e6b5 in current term 2: Already voted for candidate a5f3dd3fcdf4491ab844a80de8e30a3a in this term.
I20250626 02:01:42.640357 21949 leader_election.cc:290] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [CANDIDATE]: Term 2 election: Requested vote from peers 90b1dbff36f9442090e578971f08e6b5 (127.19.220.189:37117), c3c7ed0f056e44aeb3e500a5062fbbb8 (127.19.220.188:46299)
I20250626 02:01:42.640393 21507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "90b1dbff36f9442090e578971f08e6b5"
I20250626 02:01:42.640488 21485 leader_election.cc:304] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90b1dbff36f9442090e578971f08e6b5; no voters: a5f3dd3fcdf4491ab844a80de8e30a3a, c3c7ed0f056e44aeb3e500a5062fbbb8
I20250626 02:01:42.640666 21507 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 1 FOLLOWER]: Advancing to term 2
W20250626 02:01:42.641072 21418 leader_election.cc:336] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c3c7ed0f056e44aeb3e500a5062fbbb8 (127.19.220.188:46299): Network error: Client connection negotiation failed: client connection to 127.19.220.188:46299: connect: Connection refused (error 111)
I20250626 02:01:42.641639 21507 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f3dd3fcdf4491ab844a80de8e30a3a in term 2.
I20250626 02:01:42.641744 21950 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 02:01:42.641916 21419 leader_election.cc:304] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90b1dbff36f9442090e578971f08e6b5, a5f3dd3fcdf4491ab844a80de8e30a3a; no voters: c3c7ed0f056e44aeb3e500a5062fbbb8
I20250626 02:01:42.642073 21949 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:01:42.642217 21949 raft_consensus.cc:695] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 2 LEADER]: Becoming Leader. State: Replica: a5f3dd3fcdf4491ab844a80de8e30a3a, State: Running, Role: LEADER
I20250626 02:01:42.642354 21949 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } }
I20250626 02:01:42.642942 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f3dd3fcdf4491ab844a80de8e30a3a. Latest consensus state: current_term: 2 leader_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5f3dd3fcdf4491ab844a80de8e30a3a" member_type: VOTER last_known_addr { host: "127.19.220.190" port: 38219 } } peers { permanent_uuid: "90b1dbff36f9442090e578971f08e6b5" member_type: VOTER last_known_addr { host: "127.19.220.189" port: 37117 } } peers { permanent_uuid: "c3c7ed0f056e44aeb3e500a5062fbbb8" member_type: VOTER last_known_addr { host: "127.19.220.188" port: 46299 } } }
I20250626 02:01:42.643023 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:42.643261 21957 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:01:42.644300 21957 catalog_manager.cc:671] Loaded metadata for table test-workload [id=db68d16267da4c9a919c7ab9fe1d50ea]
I20250626 02:01:42.644980 21957 tablet_loader.cc:96] loaded metadata for tablet 668623f6f3f74deda264481fe2fb07f7 (table test-workload [id=db68d16267da4c9a919c7ab9fe1d50ea])
I20250626 02:01:42.645172 21957 tablet_loader.cc:96] loaded metadata for tablet 6c6147cb54c343bf98ae4c28786e06fe (table test-workload [id=db68d16267da4c9a919c7ab9fe1d50ea])
I20250626 02:01:42.645349 21957 tablet_loader.cc:96] loaded metadata for tablet afcfb59ea16b43cbaed8bb2d61158bc4 (table test-workload [id=db68d16267da4c9a919c7ab9fe1d50ea])
I20250626 02:01:42.645447 21957 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:01:42.645812 21957 catalog_manager.cc:1261] Loaded cluster ID: f5ad075837e0432182f32bed7dedf44b
I20250626 02:01:42.645856 21957 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:01:42.646373 21957 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:01:42.646766 21957 catalog_manager.cc:5966] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: Loaded TSK: 0
I20250626 02:01:42.647185 21957 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:01:42.878901 21616 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:42.879031 21616 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:42.919226 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:01:42.923223 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:42.923429 20338 tablet_replica.cc:331] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835: stopping tablet replica
I20250626 02:01:42.923544 20338 raft_consensus.cc:2241] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:42.923667 20338 raft_consensus.cc:2270] T 668623f6f3f74deda264481fe2fb07f7 P 3003bea79883450f819ac70c12cf3835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:42.936071 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:01:42.938025 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:01:42.941520 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:42.941696 20338 tablet_replica.cc:331] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c: stopping tablet replica
I20250626 02:01:42.941807 20338 raft_consensus.cc:2241] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:42.941905 20338 raft_consensus.cc:2270] T afcfb59ea16b43cbaed8bb2d61158bc4 P d5673596f3784121a7c55bb45c46645c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:42.954326 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:01:42.956398 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:01:42.959735 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:01:42.959918 20338 tablet_replica.cc:331] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a: stopping tablet replica
I20250626 02:01:42.960032 20338 raft_consensus.cc:2241] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:01:42.960173 20338 raft_consensus.cc:2270] T 6c6147cb54c343bf98ae4c28786e06fe P 9fb255bf645c4c51b195f3d35597e72a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:42.972643 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:01:42.974467 20338 master.cc:561] Master@127.19.220.190:38219 shutting down...
I20250626 02:01:42.977483 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:01:42.977664 20338 pending_rounds.cc:70] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: Trying to abort 1 pending ops.
I20250626 02:01:42.977697 20338 pending_rounds.cc:77] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: Aborting op as it isn't in flight: id { term: 2 index: 10 } timestamp: 7171699927624400896 op_type: NO_OP noop_request { }
I20250626 02:01:42.977777 20338 raft_consensus.cc:2887] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:01:42.977839 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:42.977949 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P a5f3dd3fcdf4491ab844a80de8e30a3a: stopping tablet replica
I20250626 02:01:42.980518 20338 master.cc:583] Master@127.19.220.190:38219 shutdown complete.
I20250626 02:01:42.982662 20338 master.cc:561] Master@127.19.220.189:37117 shutting down...
I20250626 02:01:42.985363 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:42.985489 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:42.985554 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P 90b1dbff36f9442090e578971f08e6b5: stopping tablet replica
I20250626 02:01:42.997861 20338 master.cc:583] Master@127.19.220.189:37117 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4217 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20250626 02:01:43.004758 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:34415
I20250626 02:01:43.005095 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:43.006599 21961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:43.006584 21958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:43.006685 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:43.006599 21959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:43.007122 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:43.007165 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:43.007180 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903303007181 us; error 0 us; skew 500 ppm
I20250626 02:01:43.008042 20338 webserver.cc:469] Webserver started at http://127.19.220.190:45101/ using document root <none> and password file <none>
I20250626 02:01:43.008210 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:43.008263 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:43.008318 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:43.008639 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "7569e3136373423994c239a2afa933cb"
format_stamp: "Formatted at 2025-06-26 02:01:43 on dist-test-slave-h18f"
I20250626 02:01:43.009999 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:01:43.010752 21966 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.010990 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:43.011047 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "7569e3136373423994c239a2afa933cb"
format_stamp: "Formatted at 2025-06-26 02:01:43 on dist-test-slave-h18f"
I20250626 02:01:43.011116 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:43.020226 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:43.020539 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:43.024291 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:34415
I20250626 02:01:43.027050 22018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:34415 every 8 connection(s)
I20250626 02:01:43.027446 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:43.029209 22019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb: Bootstrap starting.
I20250626 02:01:43.029932 22019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:43.030794 22019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb: No bootstrap required, opened a new log
I20250626 02:01:43.031108 22019 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569e3136373423994c239a2afa933cb" member_type: VOTER }
I20250626 02:01:43.031184 22019 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:43.031205 22019 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7569e3136373423994c239a2afa933cb, State: Initialized, Role: FOLLOWER
I20250626 02:01:43.031303 22019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569e3136373423994c239a2afa933cb" member_type: VOTER }
I20250626 02:01:43.031353 22019 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:01:43.031375 22019 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:01:43.031406 22019 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:43.032022 22019 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569e3136373423994c239a2afa933cb" member_type: VOTER }
I20250626 02:01:43.032147 22019 leader_election.cc:304] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7569e3136373423994c239a2afa933cb; no voters:
I20250626 02:01:43.032315 22019 leader_election.cc:290] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:01:43.032408 22022 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:43.032613 22022 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 1 LEADER]: Becoming Leader. State: Replica: 7569e3136373423994c239a2afa933cb, State: Running, Role: LEADER
I20250626 02:01:43.032739 22019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:01:43.032740 22022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569e3136373423994c239a2afa933cb" member_type: VOTER }
I20250626 02:01:43.033200 22023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7569e3136373423994c239a2afa933cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569e3136373423994c239a2afa933cb" member_type: VOTER } }
I20250626 02:01:43.033228 22024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7569e3136373423994c239a2afa933cb. Latest consensus state: current_term: 1 leader_uuid: "7569e3136373423994c239a2afa933cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7569e3136373423994c239a2afa933cb" member_type: VOTER } }
I20250626 02:01:43.033352 22023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:43.033371 22024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [sys.catalog]: This master's current role is: LEADER
I20250626 02:01:43.033931 22030 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:01:43.034715 22030 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:01:43.034854 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:01:43.036381 22030 catalog_manager.cc:1349] Generated new cluster ID: 008b98f7cd0a41198e1e78723e32c915
I20250626 02:01:43.036438 22030 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:01:43.051251 22030 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:01:43.051759 22030 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:01:43.065688 22030 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb: Generated new TSK 0
I20250626 02:01:43.065848 22030 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:01:43.066962 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:43.068683 22040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:43.068751 22043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:43.068866 20338 server_base.cc:1048] running on GCE node
W20250626 02:01:43.068881 22041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:43.069092 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:43.069133 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:43.069149 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903303069149 us; error 0 us; skew 500 ppm
I20250626 02:01:43.069976 20338 webserver.cc:469] Webserver started at http://127.19.220.129:36863/ using document root <none> and password file <none>
I20250626 02:01:43.070099 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:43.070142 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:43.070209 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:43.070562 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "59124e91f69e496483152936123c3ed8"
format_stamp: "Formatted at 2025-06-26 02:01:43 on dist-test-slave-h18f"
I20250626 02:01:43.071919 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:01:43.072844 22048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.073071 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:43.073129 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "59124e91f69e496483152936123c3ed8"
format_stamp: "Formatted at 2025-06-26 02:01:43 on dist-test-slave-h18f"
I20250626 02:01:43.073200 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:43.100432 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:43.100852 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:43.101151 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:43.101718 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:43.101773 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.101832 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:43.101866 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.106129 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:43315
I20250626 02:01:43.106159 22111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:43315 every 8 connection(s)
I20250626 02:01:43.106971 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:43.108860 22116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:43.108911 22117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:43.109104 22119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:43.109375 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:43.109617 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:43.109660 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:43.109678 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903303109677 us; error 0 us; skew 500 ppm
I20250626 02:01:43.110720 20338 webserver.cc:469] Webserver started at http://127.19.220.130:32995/ using document root <none> and password file <none>
I20250626 02:01:43.110890 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:43.110953 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:43.111029 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:43.111483 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "4a026f37ae434dcb9b172bb92763fc9c"
format_stamp: "Formatted at 2025-06-26 02:01:43 on dist-test-slave-h18f"
I20250626 02:01:43.111557 22112 heartbeater.cc:344] Connected to a master server at 127.19.220.190:34415
I20250626 02:01:43.111658 22112 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:43.111874 22112 heartbeater.cc:507] Master 127.19.220.190:34415 requested a full tablet report, sending...
I20250626 02:01:43.112499 21983 ts_manager.cc:194] Registered new tserver with Master: 59124e91f69e496483152936123c3ed8 (127.19.220.129:43315)
I20250626 02:01:43.113175 21983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49286
I20250626 02:01:43.113178 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:01:43.114176 22124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.114410 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:01:43.114470 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "4a026f37ae434dcb9b172bb92763fc9c"
format_stamp: "Formatted at 2025-06-26 02:01:43 on dist-test-slave-h18f"
I20250626 02:01:43.114535 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:43.148449 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:43.148886 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:43.149163 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:43.149639 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:43.149677 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.149711 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:43.149746 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.153793 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:37337
I20250626 02:01:43.153854 22187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:37337 every 8 connection(s)
I20250626 02:01:43.154572 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:43.156392 22191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:43.156494 22194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:43.156564 22192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:43.156719 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:43.156980 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:43.157034 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:43.157058 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903303157056 us; error 0 us; skew 500 ppm
I20250626 02:01:43.158185 20338 webserver.cc:469] Webserver started at http://127.19.220.131:43401/ using document root <none> and password file <none>
I20250626 02:01:43.158360 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:43.158413 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:43.158494 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:43.158900 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "2ba412835b8f469a821ff9d7a85aa7c5"
format_stamp: "Formatted at 2025-06-26 02:01:43 on dist-test-slave-h18f"
I20250626 02:01:43.159019 22188 heartbeater.cc:344] Connected to a master server at 127.19.220.190:34415
I20250626 02:01:43.159107 22188 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:43.159317 22188 heartbeater.cc:507] Master 127.19.220.190:34415 requested a full tablet report, sending...
I20250626 02:01:43.159904 21983 ts_manager.cc:194] Registered new tserver with Master: 4a026f37ae434dcb9b172bb92763fc9c (127.19.220.130:37337)
I20250626 02:01:43.160559 21983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49298
I20250626 02:01:43.160581 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:01:43.161504 22199 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.161736 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:43.161795 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "2ba412835b8f469a821ff9d7a85aa7c5"
format_stamp: "Formatted at 2025-06-26 02:01:43 on dist-test-slave-h18f"
I20250626 02:01:43.161859 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:43.192373 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:43.192760 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:43.193042 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:43.193500 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:43.193537 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.193571 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:43.193603 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:43.197850 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:44655
I20250626 02:01:43.197916 22262 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:44655 every 8 connection(s)
I20250626 02:01:43.201972 22263 heartbeater.cc:344] Connected to a master server at 127.19.220.190:34415
I20250626 02:01:43.202064 22263 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:43.202275 22263 heartbeater.cc:507] Master 127.19.220.190:34415 requested a full tablet report, sending...
I20250626 02:01:43.202821 21983 ts_manager.cc:194] Registered new tserver with Master: 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131:44655)
I20250626 02:01:43.203315 21983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49310
I20250626 02:01:43.203792 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00556937s
I20250626 02:01:44.114161 22112 heartbeater.cc:499] Master 127.19.220.190:34415 was elected leader, sending a full tablet report...
I20250626 02:01:44.161433 22188 heartbeater.cc:499] Master 127.19.220.190:34415 was elected leader, sending a full tablet report...
I20250626 02:01:44.204320 22263 heartbeater.cc:499] Master 127.19.220.190:34415 was elected leader, sending a full tablet report...
I20250626 02:01:44.230069 20338 test_util.cc:276] Using random seed: -243526478
I20250626 02:01:44.237931 21983 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49320:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:01:44.238477 21983 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:01:44.247943 22227 tablet_service.cc:1468] Processing CreateTablet for tablet cd65feed56eb4e0296b407684e837d48 (DEFAULT_TABLE table=test-workload [id=ababee6d3f7742c5a464e2740b25a072]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250626 02:01:44.248446 22226 tablet_service.cc:1468] Processing CreateTablet for tablet 9e6782ba29b6443e8275d8f403658cac (DEFAULT_TABLE table=test-workload [id=ababee6d3f7742c5a464e2740b25a072]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250626 02:01:44.248785 22075 tablet_service.cc:1468] Processing CreateTablet for tablet 9e6782ba29b6443e8275d8f403658cac (DEFAULT_TABLE table=test-workload [id=ababee6d3f7742c5a464e2740b25a072]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250626 02:01:44.249056 22075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6782ba29b6443e8275d8f403658cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:44.249037 22152 tablet_service.cc:1468] Processing CreateTablet for tablet cd65feed56eb4e0296b407684e837d48 (DEFAULT_TABLE table=test-workload [id=ababee6d3f7742c5a464e2740b25a072]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250626 02:01:44.249082 22151 tablet_service.cc:1468] Processing CreateTablet for tablet 9e6782ba29b6443e8275d8f403658cac (DEFAULT_TABLE table=test-workload [id=ababee6d3f7742c5a464e2740b25a072]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250626 02:01:44.248446 22076 tablet_service.cc:1468] Processing CreateTablet for tablet cd65feed56eb4e0296b407684e837d48 (DEFAULT_TABLE table=test-workload [id=ababee6d3f7742c5a464e2740b25a072]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250626 02:01:44.249361 22152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd65feed56eb4e0296b407684e837d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:44.249423 22076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd65feed56eb4e0296b407684e837d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:44.249529 22151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6782ba29b6443e8275d8f403658cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:44.250801 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd65feed56eb4e0296b407684e837d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:44.252077 22284 tablet_bootstrap.cc:492] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c: Bootstrap starting.
I20250626 02:01:44.252833 22226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6782ba29b6443e8275d8f403658cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:44.253098 22284 tablet_bootstrap.cc:654] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:44.253691 22285 tablet_bootstrap.cc:492] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5: Bootstrap starting.
I20250626 02:01:44.253966 22283 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8: Bootstrap starting.
I20250626 02:01:44.254256 22284 tablet_bootstrap.cc:492] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c: No bootstrap required, opened a new log
I20250626 02:01:44.254315 22284 ts_tablet_manager.cc:1397] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:44.254643 22284 raft_consensus.cc:357] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.254717 22284 raft_consensus.cc:383] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:44.254737 22284 raft_consensus.cc:738] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a026f37ae434dcb9b172bb92763fc9c, State: Initialized, Role: FOLLOWER
I20250626 02:01:44.254870 22284 consensus_queue.cc:260] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.254946 22283 tablet_bootstrap.cc:654] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:44.255232 22284 ts_tablet_manager.cc:1428] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:44.255367 22284 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c: Bootstrap starting.
I20250626 02:01:44.255944 22283 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8: No bootstrap required, opened a new log
I20250626 02:01:44.255999 22283 ts_tablet_manager.cc:1397] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:44.256314 22284 tablet_bootstrap.cc:654] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:44.256343 22283 raft_consensus.cc:357] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.256441 22283 raft_consensus.cc:383] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:44.256466 22283 raft_consensus.cc:738] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59124e91f69e496483152936123c3ed8, State: Initialized, Role: FOLLOWER
I20250626 02:01:44.256577 22283 consensus_queue.cc:260] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.256891 22283 ts_tablet_manager.cc:1428] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:44.257027 22283 tablet_bootstrap.cc:492] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8: Bootstrap starting.
I20250626 02:01:44.257586 22284 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c: No bootstrap required, opened a new log
I20250626 02:01:44.257685 22284 ts_tablet_manager.cc:1397] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:01:44.258045 22283 tablet_bootstrap.cc:654] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:44.258337 22284 raft_consensus.cc:357] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.258491 22284 raft_consensus.cc:383] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:44.258540 22284 raft_consensus.cc:738] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a026f37ae434dcb9b172bb92763fc9c, State: Initialized, Role: FOLLOWER
I20250626 02:01:44.258687 22284 consensus_queue.cc:260] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.259063 22284 ts_tablet_manager.cc:1428] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:44.259328 22283 tablet_bootstrap.cc:492] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8: No bootstrap required, opened a new log
I20250626 02:01:44.259410 22283 ts_tablet_manager.cc:1397] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:01:44.259891 22283 raft_consensus.cc:357] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.259974 22283 raft_consensus.cc:383] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:44.259987 22285 tablet_bootstrap.cc:654] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:44.260007 22283 raft_consensus.cc:738] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59124e91f69e496483152936123c3ed8, State: Initialized, Role: FOLLOWER
I20250626 02:01:44.260121 22283 consensus_queue.cc:260] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.260385 22283 ts_tablet_manager.cc:1428] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:44.261080 22285 tablet_bootstrap.cc:492] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5: No bootstrap required, opened a new log
I20250626 02:01:44.261178 22285 ts_tablet_manager.cc:1397] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250626 02:01:44.261586 22285 raft_consensus.cc:357] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.261668 22285 raft_consensus.cc:383] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:44.261699 22285 raft_consensus.cc:738] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba412835b8f469a821ff9d7a85aa7c5, State: Initialized, Role: FOLLOWER
I20250626 02:01:44.261807 22285 consensus_queue.cc:260] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.262107 22285 ts_tablet_manager.cc:1428] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:44.262262 22285 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: Bootstrap starting.
I20250626 02:01:44.263119 22285 tablet_bootstrap.cc:654] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: Neither blocks nor log segments found. Creating new log.
I20250626 02:01:44.264393 22285 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: No bootstrap required, opened a new log
I20250626 02:01:44.264467 22285 ts_tablet_manager.cc:1397] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:01:44.264791 22285 raft_consensus.cc:357] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.264869 22285 raft_consensus.cc:383] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:01:44.264906 22285 raft_consensus.cc:738] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba412835b8f469a821ff9d7a85aa7c5, State: Initialized, Role: FOLLOWER
I20250626 02:01:44.265025 22285 consensus_queue.cc:260] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.265282 22285 ts_tablet_manager.cc:1428] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:01:44.370095 22287 raft_consensus.cc:491] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:44.370241 22287 raft_consensus.cc:513] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.370762 22287 leader_election.cc:290] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59124e91f69e496483152936123c3ed8 (127.19.220.129:43315), 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131:44655)
I20250626 02:01:44.375164 22237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd65feed56eb4e0296b407684e837d48" candidate_uuid: "4a026f37ae434dcb9b172bb92763fc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" is_pre_election: true
I20250626 02:01:44.375164 22086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd65feed56eb4e0296b407684e837d48" candidate_uuid: "4a026f37ae434dcb9b172bb92763fc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59124e91f69e496483152936123c3ed8" is_pre_election: true
I20250626 02:01:44.375351 22237 raft_consensus.cc:2466] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a026f37ae434dcb9b172bb92763fc9c in term 0.
I20250626 02:01:44.375351 22086 raft_consensus.cc:2466] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a026f37ae434dcb9b172bb92763fc9c in term 0.
I20250626 02:01:44.375692 22128 leader_election.cc:304] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a026f37ae434dcb9b172bb92763fc9c, 59124e91f69e496483152936123c3ed8; no voters:
I20250626 02:01:44.375867 22287 raft_consensus.cc:2802] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:44.375931 22287 raft_consensus.cc:491] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:44.375960 22287 raft_consensus.cc:3058] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:44.376842 22287 raft_consensus.cc:513] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.377164 22287 leader_election.cc:290] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [CANDIDATE]: Term 1 election: Requested vote from peers 59124e91f69e496483152936123c3ed8 (127.19.220.129:43315), 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131:44655)
I20250626 02:01:44.377463 22086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd65feed56eb4e0296b407684e837d48" candidate_uuid: "4a026f37ae434dcb9b172bb92763fc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59124e91f69e496483152936123c3ed8"
I20250626 02:01:44.377463 22237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd65feed56eb4e0296b407684e837d48" candidate_uuid: "4a026f37ae434dcb9b172bb92763fc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba412835b8f469a821ff9d7a85aa7c5"
I20250626 02:01:44.377601 22086 raft_consensus.cc:3058] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:44.377601 22237 raft_consensus.cc:3058] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:44.378549 22086 raft_consensus.cc:2466] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a026f37ae434dcb9b172bb92763fc9c in term 1.
I20250626 02:01:44.378549 22237 raft_consensus.cc:2466] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a026f37ae434dcb9b172bb92763fc9c in term 1.
I20250626 02:01:44.378813 22128 leader_election.cc:304] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a026f37ae434dcb9b172bb92763fc9c, 59124e91f69e496483152936123c3ed8; no voters:
I20250626 02:01:44.378983 22287 raft_consensus.cc:2802] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:44.379192 22287 raft_consensus.cc:695] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 1 LEADER]: Becoming Leader. State: Replica: 4a026f37ae434dcb9b172bb92763fc9c, State: Running, Role: LEADER
I20250626 02:01:44.379333 22287 consensus_queue.cc:237] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.380690 21982 catalog_manager.cc:5582] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a026f37ae434dcb9b172bb92763fc9c (127.19.220.130). New cstate: current_term: 1 leader_uuid: "4a026f37ae434dcb9b172bb92763fc9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:44.453521 22287 raft_consensus.cc:491] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:01:44.453665 22287 raft_consensus.cc:513] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.454053 22287 leader_election.cc:290] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59124e91f69e496483152936123c3ed8 (127.19.220.129:43315), 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131:44655)
I20250626 02:01:44.454519 22237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e6782ba29b6443e8275d8f403658cac" candidate_uuid: "4a026f37ae434dcb9b172bb92763fc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" is_pre_election: true
I20250626 02:01:44.454519 22086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e6782ba29b6443e8275d8f403658cac" candidate_uuid: "4a026f37ae434dcb9b172bb92763fc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59124e91f69e496483152936123c3ed8" is_pre_election: true
I20250626 02:01:44.454695 22237 raft_consensus.cc:2466] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a026f37ae434dcb9b172bb92763fc9c in term 0.
I20250626 02:01:44.454699 22086 raft_consensus.cc:2466] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a026f37ae434dcb9b172bb92763fc9c in term 0.
I20250626 02:01:44.454988 22128 leader_election.cc:304] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a026f37ae434dcb9b172bb92763fc9c, 59124e91f69e496483152936123c3ed8; no voters:
I20250626 02:01:44.455149 22287 raft_consensus.cc:2802] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:01:44.455221 22287 raft_consensus.cc:491] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:01:44.455250 22287 raft_consensus.cc:3058] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:44.456056 22287 raft_consensus.cc:513] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.456389 22287 leader_election.cc:290] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [CANDIDATE]: Term 1 election: Requested vote from peers 59124e91f69e496483152936123c3ed8 (127.19.220.129:43315), 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131:44655)
I20250626 02:01:44.456672 22086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e6782ba29b6443e8275d8f403658cac" candidate_uuid: "4a026f37ae434dcb9b172bb92763fc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59124e91f69e496483152936123c3ed8"
I20250626 02:01:44.456718 22237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e6782ba29b6443e8275d8f403658cac" candidate_uuid: "4a026f37ae434dcb9b172bb92763fc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba412835b8f469a821ff9d7a85aa7c5"
I20250626 02:01:44.456768 22086 raft_consensus.cc:3058] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:44.456809 22237 raft_consensus.cc:3058] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:01:44.457530 22237 raft_consensus.cc:2466] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a026f37ae434dcb9b172bb92763fc9c in term 1.
I20250626 02:01:44.457533 22086 raft_consensus.cc:2466] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a026f37ae434dcb9b172bb92763fc9c in term 1.
I20250626 02:01:44.457804 22127 leader_election.cc:304] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba412835b8f469a821ff9d7a85aa7c5, 4a026f37ae434dcb9b172bb92763fc9c; no voters:
I20250626 02:01:44.458000 22287 raft_consensus.cc:2802] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:01:44.458078 22287 raft_consensus.cc:695] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 1 LEADER]: Becoming Leader. State: Replica: 4a026f37ae434dcb9b172bb92763fc9c, State: Running, Role: LEADER
I20250626 02:01:44.458215 22287 consensus_queue.cc:237] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:44.459638 21982 catalog_manager.cc:5582] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a026f37ae434dcb9b172bb92763fc9c (127.19.220.130). New cstate: current_term: 1 leader_uuid: "4a026f37ae434dcb9b172bb92763fc9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:44.794032 22287 consensus_queue.cc:1035] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:44.796844 22303 consensus_queue.cc:1035] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:44.991703 22303 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:44.994388 22297 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:01:45.034672 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:45.038767 22162 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9e6782ba29b6443e8275d8f403658cac"
dest_uuid: "4a026f37ae434dcb9b172bb92763fc9c"
mode: GRACEFUL
new_leader_uuid: "59124e91f69e496483152936123c3ed8"
from {username='slave'} at 127.0.0.1:41674
I20250626 02:01:45.038893 22162 raft_consensus.cc:604] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 1 LEADER]: Received request to transfer leadership to TS 59124e91f69e496483152936123c3ed8
I20250626 02:01:45.039254 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:01:45.039350 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:45.443626 22297 raft_consensus.cc:991] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c: : Instructing follower 59124e91f69e496483152936123c3ed8 to start an election
I20250626 02:01:45.443746 22297 raft_consensus.cc:1079] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 1 LEADER]: Signalling peer 59124e91f69e496483152936123c3ed8 to start an election
I20250626 02:01:45.444234 22086 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9e6782ba29b6443e8275d8f403658cac"
dest_uuid: "59124e91f69e496483152936123c3ed8"
from {username='slave'} at 127.0.0.1:59506
I20250626 02:01:45.444381 22086 raft_consensus.cc:491] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:01:45.444456 22086 raft_consensus.cc:3058] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:45.445312 22086 raft_consensus.cc:513] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:45.445789 22086 leader_election.cc:290] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [CANDIDATE]: Term 2 election: Requested vote from peers 4a026f37ae434dcb9b172bb92763fc9c (127.19.220.130:37337), 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131:44655)
I20250626 02:01:45.449155 22162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e6782ba29b6443e8275d8f403658cac" candidate_uuid: "59124e91f69e496483152936123c3ed8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a026f37ae434dcb9b172bb92763fc9c"
I20250626 02:01:45.449162 22237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e6782ba29b6443e8275d8f403658cac" candidate_uuid: "59124e91f69e496483152936123c3ed8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ba412835b8f469a821ff9d7a85aa7c5"
I20250626 02:01:45.449290 22162 raft_consensus.cc:3053] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:01:45.449292 22237 raft_consensus.cc:3058] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:45.449359 22162 raft_consensus.cc:738] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a026f37ae434dcb9b172bb92763fc9c, State: Running, Role: LEADER
I20250626 02:01:45.449541 22162 consensus_queue.cc:260] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:45.449698 22162 raft_consensus.cc:3058] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:01:45.450160 22237 raft_consensus.cc:2466] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59124e91f69e496483152936123c3ed8 in term 2.
I20250626 02:01:45.450456 22051 leader_election.cc:304] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba412835b8f469a821ff9d7a85aa7c5, 59124e91f69e496483152936123c3ed8; no voters:
I20250626 02:01:45.450534 22162 raft_consensus.cc:2466] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59124e91f69e496483152936123c3ed8 in term 2.
I20250626 02:01:45.450628 22300 raft_consensus.cc:2802] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:01:45.450831 22300 raft_consensus.cc:695] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 2 LEADER]: Becoming Leader. State: Replica: 59124e91f69e496483152936123c3ed8, State: Running, Role: LEADER
I20250626 02:01:45.450959 22300 consensus_queue.cc:237] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } }
I20250626 02:01:45.452313 21982 catalog_manager.cc:5582] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 reported cstate change: term changed from 1 to 2, leader changed from 4a026f37ae434dcb9b172bb92763fc9c (127.19.220.130) to 59124e91f69e496483152936123c3ed8 (127.19.220.129). New cstate: current_term: 2 leader_uuid: "59124e91f69e496483152936123c3ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } health_report { overall_health: UNKNOWN } } }
I20250626 02:01:46.024897 22162 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:01:46.025439 22320 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:46.028184 22237 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 2 FOLLOWER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:01:46.028559 22322 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:01:47.039564 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:47.039896 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:47.039945 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:49.040211 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:49.040527 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:49.040586 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:51.040813 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:51.041118 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:51.041185 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:53.041385 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:53.041759 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:53.041839 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:55.042127 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:55.042493 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:55.042544 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:55.527899 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:01:55.529678 22331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:55.529722 22334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:01:55.529695 22332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:01:55.529824 20338 server_base.cc:1048] running on GCE node
I20250626 02:01:55.530067 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:01:55.530108 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:01:55.530123 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903315530123 us; error 0 us; skew 500 ppm
I20250626 02:01:55.536427 20338 webserver.cc:469] Webserver started at http://127.19.220.132:37201/ using document root <none> and password file <none>
I20250626 02:01:55.536585 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:01:55.536651 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:01:55.536733 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:01:55.537128 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-3-root/instance:
uuid: "a704fb20d1b04d08bb854c5c940c8a9a"
format_stamp: "Formatted at 2025-06-26 02:01:55 on dist-test-slave-h18f"
I20250626 02:01:55.538515 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:01:55.539366 22339 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:55.539615 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:01:55.539673 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-3-root
uuid: "a704fb20d1b04d08bb854c5c940c8a9a"
format_stamp: "Formatted at 2025-06-26 02:01:55 on dist-test-slave-h18f"
I20250626 02:01:55.539738 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:01:55.544999 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:01:55.545327 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:01:55.545565 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:01:55.545974 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:01:55.546010 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:55.546042 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:01:55.546072 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:01:55.550671 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.132:41801
I20250626 02:01:55.550727 22403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.132:41801 every 8 connection(s)
I20250626 02:01:55.554719 22404 heartbeater.cc:344] Connected to a master server at 127.19.220.190:34415
I20250626 02:01:55.554807 22404 heartbeater.cc:461] Registering TS with master...
I20250626 02:01:55.554996 22404 heartbeater.cc:507] Master 127.19.220.190:34415 requested a full tablet report, sending...
I20250626 02:01:55.555546 21982 ts_manager.cc:194] Registered new tserver with Master: a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132:41801)
I20250626 02:01:55.556056 21982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45282
I20250626 02:01:56.049180 22086 consensus_queue.cc:237] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: true } }
I20250626 02:01:56.050563 22162 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 02:01:56.050868 22237 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 2 FOLLOWER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 02:01:56.050957 22359 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:01:56.051280 22412 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:01:56.052595 22411 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: true } } }
I20250626 02:01:56.053102 22162 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: true } } }
I20250626 02:01:56.055164 21982 catalog_manager.cc:5582] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 reported cstate change: config changed from index -1 to 3, NON_VOTER a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) added. New cstate: current_term: 2 leader_uuid: "59124e91f69e496483152936123c3ed8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 02:01:56.055956 22051 consensus_peers.cc:487] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 -> Peer a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132:41801): Couldn't send request to peer a704fb20d1b04d08bb854c5c940c8a9a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e6782ba29b6443e8275d8f403658cac. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:01:56.056226 22237 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: true } } }
I20250626 02:01:56.556818 22404 heartbeater.cc:499] Master 127.19.220.190:34415 was elected leader, sending a full tablet report...
I20250626 02:01:56.659900 22422 ts_tablet_manager.cc:927] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: Initiating tablet copy from peer 59124e91f69e496483152936123c3ed8 (127.19.220.129:43315)
I20250626 02:01:56.660280 22422 tablet_copy_client.cc:323] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:43315
I20250626 02:01:56.664187 22096 tablet_copy_service.cc:140] P 59124e91f69e496483152936123c3ed8: Received BeginTabletCopySession request for tablet 9e6782ba29b6443e8275d8f403658cac from peer a704fb20d1b04d08bb854c5c940c8a9a ({username='slave'} at 127.0.0.1:34220)
I20250626 02:01:56.664319 22096 tablet_copy_service.cc:161] P 59124e91f69e496483152936123c3ed8: Beginning new tablet copy session on tablet 9e6782ba29b6443e8275d8f403658cac from peer a704fb20d1b04d08bb854c5c940c8a9a at {username='slave'} at 127.0.0.1:34220: session id = a704fb20d1b04d08bb854c5c940c8a9a-9e6782ba29b6443e8275d8f403658cac
I20250626 02:01:56.665068 22096 tablet_copy_source_session.cc:215] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:01:56.665838 22422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6782ba29b6443e8275d8f403658cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:01:56.667956 22422 tablet_copy_client.cc:806] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: tablet copy: Starting download of 0 data blocks...
I20250626 02:01:56.668108 22422 tablet_copy_client.cc:670] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: tablet copy: Starting download of 1 WAL segments...
I20250626 02:01:56.669102 22422 tablet_copy_client.cc:538] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:01:56.670373 22422 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: Bootstrap starting.
I20250626 02:01:56.674546 22422 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:01:56.674813 22422 tablet_bootstrap.cc:492] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: Bootstrap complete.
I20250626 02:01:56.674974 22422 ts_tablet_manager.cc:1397] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250626 02:01:56.675448 22422 raft_consensus.cc:357] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: true } }
I20250626 02:01:56.675576 22422 raft_consensus.cc:738] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a704fb20d1b04d08bb854c5c940c8a9a, State: Initialized, Role: LEARNER
I20250626 02:01:56.675714 22422 consensus_queue.cc:260] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: true } }
I20250626 02:01:56.676069 22422 ts_tablet_manager.cc:1428] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20250626 02:01:56.676529 22096 tablet_copy_service.cc:342] P 59124e91f69e496483152936123c3ed8: Request end of tablet copy session a704fb20d1b04d08bb854c5c940c8a9a-9e6782ba29b6443e8275d8f403658cac received from {username='slave'} at 127.0.0.1:34220
I20250626 02:01:56.676604 22096 tablet_copy_service.cc:434] P 59124e91f69e496483152936123c3ed8: ending tablet copy session a704fb20d1b04d08bb854c5c940c8a9a-9e6782ba29b6443e8275d8f403658cac on tablet 9e6782ba29b6443e8275d8f403658cac with peer a704fb20d1b04d08bb854c5c940c8a9a
I20250626 02:01:57.042757 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:57.043118 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:57.043191 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:01:57.179863 22378 raft_consensus.cc:1215] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250626 02:01:57.719216 22412 raft_consensus.cc:1062] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8: attempting to promote NON_VOTER a704fb20d1b04d08bb854c5c940c8a9a to VOTER
I20250626 02:01:57.719769 22412 consensus_queue.cc:237] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } }
I20250626 02:01:57.721470 22162 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 02:01:57.721482 22237 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 2 FOLLOWER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 02:01:57.721985 22378 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 LEARNER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 02:01:57.722163 22430 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:01:57.722342 22412 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:01:57.722627 22412 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:01:57.725126 22433 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } } }
I20250626 02:01:57.725504 22237 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } } }
I20250626 02:01:57.726615 22162 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } } }
I20250626 02:01:57.815325 22378 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } } }
I20250626 02:01:57.815587 21983 catalog_manager.cc:5582] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c reported cstate change: config changed from index 3 to 4, a704fb20d1b04d08bb854c5c940c8a9a (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "59124e91f69e496483152936123c3ed8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "2ba412835b8f469a821ff9d7a85aa7c5" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } } }
I20250626 02:01:57.821372 22086 consensus_queue.cc:237] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } }
I20250626 02:01:57.822451 22378 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 FOLLOWER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 02:01:57.822630 22162 raft_consensus.cc:1273] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Refusing update from remote peer 59124e91f69e496483152936123c3ed8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 02:01:57.822779 22433 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:01:57.822934 22412 consensus_queue.cc:1035] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:01:57.824397 22433 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } } }
I20250626 02:01:57.824810 22378 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } } }
I20250626 02:01:57.825744 21970 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9e6782ba29b6443e8275d8f403658cac with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:01:57.825860 22162 raft_consensus.cc:2953] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } } }
I20250626 02:01:57.826455 21980 catalog_manager.cc:5582] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 reported cstate change: config changed from index 4 to 5, VOTER 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131) evicted. New cstate: current_term: 2 leader_uuid: "59124e91f69e496483152936123c3ed8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59124e91f69e496483152936123c3ed8" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a026f37ae434dcb9b172bb92763fc9c" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 37337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a704fb20d1b04d08bb854c5c940c8a9a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:01:57.828900 22226 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e6782ba29b6443e8275d8f403658cac with delete_type TABLET_DATA_TOMBSTONED (TS 2ba412835b8f469a821ff9d7a85aa7c5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58488
I20250626 02:01:57.829437 22443 tablet_replica.cc:331] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: stopping tablet replica
I20250626 02:01:57.829569 22443 raft_consensus.cc:2241] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:01:57.829710 22443 raft_consensus.cc:2270] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:01:57.830286 22443 ts_tablet_manager.cc:1905] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:01:57.832003 22443 ts_tablet_manager.cc:1918] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250626 02:01:57.832078 22443 log.cc:1199] T 9e6782ba29b6443e8275d8f403658cac P 2ba412835b8f469a821ff9d7a85aa7c5: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/9e6782ba29b6443e8275d8f403658cac
I20250626 02:01:57.832581 21969 catalog_manager.cc:4928] TS 2ba412835b8f469a821ff9d7a85aa7c5 (127.19.220.131:44655): tablet 9e6782ba29b6443e8275d8f403658cac (table test-workload [id=ababee6d3f7742c5a464e2740b25a072]) successfully deleted
I20250626 02:01:59.043491 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:01:59.043901 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:01:59.043972 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:01.044238 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:01.044636 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:01.044703 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:03.045059 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:03.045491 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:03.045583 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:05.045928 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:05.046347 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:05.046420 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:07.046747 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:07.047150 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:07.047216 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:09.047542 22037 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:09.047966 22037 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:09.048034 22037 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:09.146201 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:02:09.150170 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:09.150393 20338 tablet_replica.cc:331] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8: stopping tablet replica
I20250626 02:02:09.150527 20338 raft_consensus.cc:2241] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:02:09.150664 20338 raft_consensus.cc:2270] T cd65feed56eb4e0296b407684e837d48 P 59124e91f69e496483152936123c3ed8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:09.151120 20338 tablet_replica.cc:331] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8: stopping tablet replica
I20250626 02:02:09.151204 20338 raft_consensus.cc:2241] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:02:09.151402 20338 raft_consensus.cc:2270] T 9e6782ba29b6443e8275d8f403658cac P 59124e91f69e496483152936123c3ed8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:09.164464 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:02:09.166447 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:02:09.171170 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:09.171350 20338 tablet_replica.cc:331] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c: stopping tablet replica
I20250626 02:02:09.171478 20338 raft_consensus.cc:2241] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:02:09.171609 20338 raft_consensus.cc:2270] T 9e6782ba29b6443e8275d8f403658cac P 4a026f37ae434dcb9b172bb92763fc9c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:09.172003 20338 tablet_replica.cc:331] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c: stopping tablet replica
I20250626 02:02:09.172072 20338 raft_consensus.cc:2241] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:02:09.172303 20338 raft_consensus.cc:2270] T cd65feed56eb4e0296b407684e837d48 P 4a026f37ae434dcb9b172bb92763fc9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:09.184978 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:02:09.187036 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:02:09.191385 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:09.191581 20338 tablet_replica.cc:331] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5: stopping tablet replica
I20250626 02:02:09.191686 20338 raft_consensus.cc:2241] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:02:09.191784 20338 raft_consensus.cc:2270] T cd65feed56eb4e0296b407684e837d48 P 2ba412835b8f469a821ff9d7a85aa7c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:09.205206 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:02:09.207093 20338 tablet_server.cc:178] TabletServer@127.19.220.132:0 shutting down...
I20250626 02:02:09.211086 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:09.211257 20338 tablet_replica.cc:331] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a: stopping tablet replica
I20250626 02:02:09.211386 20338 raft_consensus.cc:2241] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:02:09.211520 20338 raft_consensus.cc:2270] T 9e6782ba29b6443e8275d8f403658cac P a704fb20d1b04d08bb854c5c940c8a9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:09.223914 20338 tablet_server.cc:195] TabletServer@127.19.220.132:0 shutdown complete.
I20250626 02:02:09.225723 20338 master.cc:561] Master@127.19.220.190:34415 shutting down...
I20250626 02:02:09.229372 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:02:09.229503 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:09.229573 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7569e3136373423994c239a2afa933cb: stopping tablet replica
I20250626 02:02:09.241868 20338 master.cc:583] Master@127.19.220.190:34415 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26243 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20250626 02:02:09.248721 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:44281
I20250626 02:02:09.248999 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:09.250505 22449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:09.250633 22447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:09.250703 22446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:09.251001 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:09.251116 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:09.251148 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:09.251175 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903329251175 us; error 0 us; skew 500 ppm
I20250626 02:02:09.258095 20338 webserver.cc:469] Webserver started at http://127.19.220.190:43899/ using document root <none> and password file <none>
I20250626 02:02:09.258229 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:09.258270 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:09.258337 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:09.258667 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "e0112f0712984d23b74a979713022006"
format_stamp: "Formatted at 2025-06-26 02:02:09 on dist-test-slave-h18f"
I20250626 02:02:09.260043 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:09.260983 22454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:09.261247 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:09.261308 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "e0112f0712984d23b74a979713022006"
format_stamp: "Formatted at 2025-06-26 02:02:09 on dist-test-slave-h18f"
I20250626 02:02:09.261374 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:09.271322 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:09.271616 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:09.275501 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:44281
I20250626 02:02:09.279001 22507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:09.280210 22506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:44281 every 8 connection(s)
I20250626 02:02:09.280892 22507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006: Bootstrap starting.
I20250626 02:02:09.281662 22507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:09.282573 22507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006: No bootstrap required, opened a new log
I20250626 02:02:09.282905 22507 raft_consensus.cc:357] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0112f0712984d23b74a979713022006" member_type: VOTER }
I20250626 02:02:09.282985 22507 raft_consensus.cc:383] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:09.283018 22507 raft_consensus.cc:738] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0112f0712984d23b74a979713022006, State: Initialized, Role: FOLLOWER
I20250626 02:02:09.283164 22507 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0112f0712984d23b74a979713022006" member_type: VOTER }
I20250626 02:02:09.283229 22507 raft_consensus.cc:397] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:02:09.283277 22507 raft_consensus.cc:491] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:02:09.283334 22507 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:09.283947 22507 raft_consensus.cc:513] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0112f0712984d23b74a979713022006" member_type: VOTER }
I20250626 02:02:09.284065 22507 leader_election.cc:304] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0112f0712984d23b74a979713022006; no voters:
I20250626 02:02:09.284250 22507 leader_election.cc:290] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:02:09.284353 22510 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:02:09.284551 22510 raft_consensus.cc:695] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 1 LEADER]: Becoming Leader. State: Replica: e0112f0712984d23b74a979713022006, State: Running, Role: LEADER
I20250626 02:02:09.284636 22507 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:02:09.284674 22510 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0112f0712984d23b74a979713022006" member_type: VOTER }
I20250626 02:02:09.285082 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0112f0712984d23b74a979713022006. Latest consensus state: current_term: 1 leader_uuid: "e0112f0712984d23b74a979713022006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0112f0712984d23b74a979713022006" member_type: VOTER } }
I20250626 02:02:09.285204 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:09.285070 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0112f0712984d23b74a979713022006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0112f0712984d23b74a979713022006" member_type: VOTER } }
I20250626 02:02:09.285463 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:09.285884 22515 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:02:09.286847 22515 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:02:09.287050 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:02:09.288512 22515 catalog_manager.cc:1349] Generated new cluster ID: 17adf7e8b39c46bd92829988be03b4de
I20250626 02:02:09.288568 22515 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:02:09.315583 22515 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:02:09.316164 22515 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:02:09.325062 22515 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006: Generated new TSK 0
I20250626 02:02:09.325222 22515 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:02:09.351073 20338 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 7.1098e-05s
I20250626 02:02:21.430085 20338 master.cc:561] Master@127.19.220.190:44281 shutting down...
I20250626 02:02:21.433053 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:02:21.433224 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:21.433295 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P e0112f0712984d23b74a979713022006: stopping tablet replica
I20250626 02:02:21.445432 20338 master.cc:583] Master@127.19.220.190:44281 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12200 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20250626 02:02:21.449462 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:37375
I20250626 02:02:21.449791 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:21.451337 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:21.451418 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:21.451452 20338 server_base.cc:1048] running on GCE node
W20250626 02:02:21.451373 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:21.451754 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:21.451797 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:21.451821 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903341451820 us; error 0 us; skew 500 ppm
I20250626 02:02:21.452718 20338 webserver.cc:469] Webserver started at http://127.19.220.190:43249/ using document root <none> and password file <none>
I20250626 02:02:21.452859 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:21.452910 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:21.452987 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:21.453344 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "118463b838c94a35907b272b710916a1"
format_stamp: "Formatted at 2025-06-26 02:02:21 on dist-test-slave-h18f"
I20250626 02:02:21.454689 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:21.455479 22537 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.455708 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:21.455765 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "118463b838c94a35907b272b710916a1"
format_stamp: "Formatted at 2025-06-26 02:02:21 on dist-test-slave-h18f"
I20250626 02:02:21.455816 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:21.480490 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:21.480850 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:21.484827 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:37375
I20250626 02:02:21.485538 22589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:37375 every 8 connection(s)
I20250626 02:02:21.486029 22590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:21.487782 22590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1: Bootstrap starting.
I20250626 02:02:21.488569 22590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:21.489449 22590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1: No bootstrap required, opened a new log
I20250626 02:02:21.489778 22590 raft_consensus.cc:357] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118463b838c94a35907b272b710916a1" member_type: VOTER }
I20250626 02:02:21.489863 22590 raft_consensus.cc:383] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:21.489899 22590 raft_consensus.cc:738] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118463b838c94a35907b272b710916a1, State: Initialized, Role: FOLLOWER
I20250626 02:02:21.490020 22590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118463b838c94a35907b272b710916a1" member_type: VOTER }
I20250626 02:02:21.490082 22590 raft_consensus.cc:397] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:02:21.490123 22590 raft_consensus.cc:491] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:02:21.490170 22590 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:21.490828 22590 raft_consensus.cc:513] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118463b838c94a35907b272b710916a1" member_type: VOTER }
I20250626 02:02:21.490939 22590 leader_election.cc:304] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118463b838c94a35907b272b710916a1; no voters:
I20250626 02:02:21.491098 22590 leader_election.cc:290] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:02:21.491173 22593 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:02:21.491390 22593 raft_consensus.cc:695] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 1 LEADER]: Becoming Leader. State: Replica: 118463b838c94a35907b272b710916a1, State: Running, Role: LEADER
I20250626 02:02:21.491511 22590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:02:21.491515 22593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118463b838c94a35907b272b710916a1" member_type: VOTER }
I20250626 02:02:21.491945 22594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "118463b838c94a35907b272b710916a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118463b838c94a35907b272b710916a1" member_type: VOTER } }
I20250626 02:02:21.491969 22595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118463b838c94a35907b272b710916a1. Latest consensus state: current_term: 1 leader_uuid: "118463b838c94a35907b272b710916a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118463b838c94a35907b272b710916a1" member_type: VOTER } }
I20250626 02:02:21.492097 22594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:21.492148 22595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:21.492602 22598 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:02:21.493572 22598 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:02:21.493780 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:02:21.495278 22598 catalog_manager.cc:1349] Generated new cluster ID: 14b7c7cb9cab49cba5af889b3f70d34c
I20250626 02:02:21.495340 22598 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:02:21.503505 22598 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:02:21.504034 22598 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:02:21.510056 22598 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1: Generated new TSK 0
I20250626 02:02:21.510202 22598 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:02:21.526109 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:21.527899 22611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:21.528031 20338 server_base.cc:1048] running on GCE node
W20250626 02:02:21.527992 22612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:21.527992 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:21.528323 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:21.528366 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:21.528381 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903341528381 us; error 0 us; skew 500 ppm
I20250626 02:02:21.529253 20338 webserver.cc:469] Webserver started at http://127.19.220.129:41721/ using document root <none> and password file <none>
I20250626 02:02:21.529395 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:21.529435 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:21.529489 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:21.529815 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "e030d44e4f5f431e866624cc066bfc0e"
format_stamp: "Formatted at 2025-06-26 02:02:21 on dist-test-slave-h18f"
I20250626 02:02:21.531075 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:02:21.531842 22619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.532047 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:02:21.532104 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "e030d44e4f5f431e866624cc066bfc0e"
format_stamp: "Formatted at 2025-06-26 02:02:21 on dist-test-slave-h18f"
I20250626 02:02:21.532191 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:21.546906 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:21.547326 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:21.547626 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:21.548101 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:21.548211 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.548269 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:21.548300 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.552307 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:44415
I20250626 02:02:21.552515 22682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:44415 every 8 connection(s)
I20250626 02:02:21.553298 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:21.554947 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:21.555056 20338 server_base.cc:1048] running on GCE node
W20250626 02:02:21.555109 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:21.555564 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:21.555757 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:21.555799 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:21.555815 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903341555815 us; error 0 us; skew 500 ppm
I20250626 02:02:21.556916 20338 webserver.cc:469] Webserver started at http://127.19.220.130:42553/ using document root <none> and password file <none>
I20250626 02:02:21.557083 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:21.557142 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:21.557230 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:21.557723 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "1bfb03ed231a4e4faf746f5043b783e8"
format_stamp: "Formatted at 2025-06-26 02:02:21 on dist-test-slave-h18f"
I20250626 02:02:21.558085 22683 heartbeater.cc:344] Connected to a master server at 127.19.220.190:37375
I20250626 02:02:21.558182 22683 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:21.558439 22683 heartbeater.cc:507] Master 127.19.220.190:37375 requested a full tablet report, sending...
I20250626 02:02:21.559054 22554 ts_manager.cc:194] Registered new tserver with Master: e030d44e4f5f431e866624cc066bfc0e (127.19.220.129:44415)
I20250626 02:02:21.559464 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 02:02:21.559847 22554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50274
I20250626 02:02:21.560385 22695 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.560628 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:21.560688 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "1bfb03ed231a4e4faf746f5043b783e8"
format_stamp: "Formatted at 2025-06-26 02:02:21 on dist-test-slave-h18f"
I20250626 02:02:21.560756 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:21.584784 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:21.585199 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:21.585521 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:21.585994 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:21.586035 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.586081 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:21.586112 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.590045 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:37371
I20250626 02:02:21.590109 22758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:37371 every 8 connection(s)
I20250626 02:02:21.590912 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:21.592823 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:21.592846 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:21.592900 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:21.593091 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:21.593253 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:21.593297 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:21.593318 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903341593318 us; error 0 us; skew 500 ppm
I20250626 02:02:21.594420 20338 webserver.cc:469] Webserver started at http://127.19.220.131:33537/ using document root <none> and password file <none>
I20250626 02:02:21.594587 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:21.594635 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:21.594705 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:21.595141 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "75bf87405a3e4f4d80729b8d90ecca17"
format_stamp: "Formatted at 2025-06-26 02:02:21 on dist-test-slave-h18f"
I20250626 02:02:21.595191 22759 heartbeater.cc:344] Connected to a master server at 127.19.220.190:37375
I20250626 02:02:21.595266 22759 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:21.595430 22759 heartbeater.cc:507] Master 127.19.220.190:37375 requested a full tablet report, sending...
I20250626 02:02:21.596028 22554 ts_manager.cc:194] Registered new tserver with Master: 1bfb03ed231a4e4faf746f5043b783e8 (127.19.220.130:37371)
I20250626 02:02:21.596724 22554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50278
I20250626 02:02:21.596849 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:02:21.597769 22770 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.597990 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:21.598045 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "75bf87405a3e4f4d80729b8d90ecca17"
format_stamp: "Formatted at 2025-06-26 02:02:21 on dist-test-slave-h18f"
I20250626 02:02:21.598111 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:21.620836 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:21.621269 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:21.621582 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:21.622030 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:21.622071 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.622118 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:21.622151 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:21.626581 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:36247
I20250626 02:02:21.626644 22833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:36247 every 8 connection(s)
I20250626 02:02:21.630884 22834 heartbeater.cc:344] Connected to a master server at 127.19.220.190:37375
I20250626 02:02:21.630963 22834 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:21.631130 22834 heartbeater.cc:507] Master 127.19.220.190:37375 requested a full tablet report, sending...
I20250626 02:02:21.631666 22554 ts_manager.cc:194] Registered new tserver with Master: 75bf87405a3e4f4d80729b8d90ecca17 (127.19.220.131:36247)
I20250626 02:02:21.632154 22554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50290
I20250626 02:02:21.632594 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005620547s
I20250626 02:02:22.560784 22683 heartbeater.cc:499] Master 127.19.220.190:37375 was elected leader, sending a full tablet report...
I20250626 02:02:22.597582 22759 heartbeater.cc:499] Master 127.19.220.190:37375 was elected leader, sending a full tablet report...
I20250626 02:02:22.633029 22834 heartbeater.cc:499] Master 127.19.220.190:37375 was elected leader, sending a full tablet report...
I20250626 02:02:23.493649 22608 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:25.494009 22608 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:27.494428 22608 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:29.494791 22608 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:31.495152 22608 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:33.495541 22608 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:33.714109 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:02:33.717856 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:33.730795 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:02:33.732792 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:02:33.736645 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:33.748518 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:02:33.750223 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:02:33.754416 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:33.766165 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:02:33.767916 20338 master.cc:561] Master@127.19.220.190:37375 shutting down...
I20250626 02:02:33.771541 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:02:33.771696 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:33.771761 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P 118463b838c94a35907b272b710916a1: stopping tablet replica
I20250626 02:02:33.784197 20338 master.cc:583] Master@127.19.220.190:37375 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12340 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20250626 02:02:33.789780 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:41307
I20250626 02:02:33.790107 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:33.791546 22839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.791605 22841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.791777 22838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:33.791803 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:33.792039 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:33.792080 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:33.792095 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903353792095 us; error 0 us; skew 500 ppm
I20250626 02:02:33.792979 20338 webserver.cc:469] Webserver started at http://127.19.220.190:35437/ using document root <none> and password file <none>
I20250626 02:02:33.793102 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:33.793143 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:33.793196 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:33.793570 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "6536e132c2e8422aa0a153dd45bb8188"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.794896 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:02:33.795691 22846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.795917 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:33.795980 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "6536e132c2e8422aa0a153dd45bb8188"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.796051 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:33.805133 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:33.805506 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:33.809373 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:41307
I20250626 02:02:33.810957 22898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:41307 every 8 connection(s)
I20250626 02:02:33.811359 22899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:33.813220 22899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188: Bootstrap starting.
I20250626 02:02:33.813946 22899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:33.814824 22899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188: No bootstrap required, opened a new log
I20250626 02:02:33.815137 22899 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6536e132c2e8422aa0a153dd45bb8188" member_type: VOTER }
I20250626 02:02:33.815215 22899 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:33.815243 22899 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6536e132c2e8422aa0a153dd45bb8188, State: Initialized, Role: FOLLOWER
I20250626 02:02:33.815348 22899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6536e132c2e8422aa0a153dd45bb8188" member_type: VOTER }
I20250626 02:02:33.815433 22899 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:02:33.815482 22899 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:02:33.815512 22899 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:33.816112 22899 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6536e132c2e8422aa0a153dd45bb8188" member_type: VOTER }
I20250626 02:02:33.816267 22899 leader_election.cc:304] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6536e132c2e8422aa0a153dd45bb8188; no voters:
I20250626 02:02:33.816414 22899 leader_election.cc:290] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:02:33.816540 22902 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:02:33.816727 22902 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 1 LEADER]: Becoming Leader. State: Replica: 6536e132c2e8422aa0a153dd45bb8188, State: Running, Role: LEADER
I20250626 02:02:33.816893 22899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:02:33.816881 22902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6536e132c2e8422aa0a153dd45bb8188" member_type: VOTER }
I20250626 02:02:33.817404 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6536e132c2e8422aa0a153dd45bb8188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6536e132c2e8422aa0a153dd45bb8188" member_type: VOTER } }
I20250626 02:02:33.817430 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6536e132c2e8422aa0a153dd45bb8188. Latest consensus state: current_term: 1 leader_uuid: "6536e132c2e8422aa0a153dd45bb8188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6536e132c2e8422aa0a153dd45bb8188" member_type: VOTER } }
I20250626 02:02:33.817659 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:33.817560 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:33.818176 22909 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:02:33.818948 22909 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:02:33.819126 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:02:33.820724 22909 catalog_manager.cc:1349] Generated new cluster ID: cb3bb2de8f3f454ead630e1d93f493ee
I20250626 02:02:33.820782 22909 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:02:33.840022 22909 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:02:33.840620 22909 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:02:33.854146 22909 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188: Generated new TSK 0
I20250626 02:02:33.854338 22909 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:02:33.883891 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:33.885869 22923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.885970 22920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.885998 22921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:33.886057 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:33.886358 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:33.886410 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:33.886427 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903353886427 us; error 0 us; skew 500 ppm
I20250626 02:02:33.887336 20338 webserver.cc:469] Webserver started at http://127.19.220.129:42183/ using document root <none> and password file <none>
I20250626 02:02:33.887485 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:33.887529 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:33.887588 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:33.887959 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "e5720dbd9c3145ebad87c0215e7ad424"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.889448 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:02:33.890499 22928 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.890756 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:02:33.890815 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "e5720dbd9c3145ebad87c0215e7ad424"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.890897 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:33.905407 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:33.905817 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:33.906083 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:33.906517 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:33.906558 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.906606 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:33.906643 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.910638 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:36747
I20250626 02:02:33.910669 22991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:36747 every 8 connection(s)
I20250626 02:02:33.911410 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:33.913122 22996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.913224 22999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.913249 22997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:33.913519 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:33.913717 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:33.913775 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:33.913808 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903353913807 us; error 0 us; skew 500 ppm
I20250626 02:02:33.914861 20338 webserver.cc:469] Webserver started at http://127.19.220.130:45311/ using document root <none> and password file <none>
I20250626 02:02:33.915027 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:33.915081 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:33.915153 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:33.915560 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "b2fbfc66d7af4defb8f6281a55d9f698"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.915726 22992 heartbeater.cc:344] Connected to a master server at 127.19.220.190:41307
I20250626 02:02:33.915800 22992 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:33.915967 22992 heartbeater.cc:507] Master 127.19.220.190:41307 requested a full tablet report, sending...
I20250626 02:02:33.916530 22863 ts_manager.cc:194] Registered new tserver with Master: e5720dbd9c3145ebad87c0215e7ad424 (127.19.220.129:36747)
I20250626 02:02:33.917102 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:02:33.917364 22863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39862
I20250626 02:02:33.918035 23004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.918277 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:02:33.918339 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "b2fbfc66d7af4defb8f6281a55d9f698"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.918421 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:33.933291 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:33.933710 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:33.933990 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:33.934463 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:33.934500 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.934551 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:33.934577 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.938702 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:36929
I20250626 02:02:33.938767 23067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:36929 every 8 connection(s)
I20250626 02:02:33.939467 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:33.941406 23072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.941471 23071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.941545 23074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:33.941743 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:33.941913 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:33.941952 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:33.941977 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903353941976 us; error 0 us; skew 500 ppm
I20250626 02:02:33.942994 20338 webserver.cc:469] Webserver started at http://127.19.220.131:40799/ using document root <none> and password file <none>
I20250626 02:02:33.943203 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:33.943285 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:33.943369 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:33.943828 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "f3eb450a02d44ab78801e13fb092fef6"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.943881 23068 heartbeater.cc:344] Connected to a master server at 127.19.220.190:41307
I20250626 02:02:33.943979 23068 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:33.944231 23068 heartbeater.cc:507] Master 127.19.220.190:41307 requested a full tablet report, sending...
I20250626 02:02:33.944869 22863 ts_manager.cc:194] Registered new tserver with Master: b2fbfc66d7af4defb8f6281a55d9f698 (127.19.220.130:36929)
I20250626 02:02:33.945384 22863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39878
I20250626 02:02:33.945462 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:02:33.946413 23079 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.946705 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:33.946774 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "f3eb450a02d44ab78801e13fb092fef6"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.946828 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:33.958714 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:33.959115 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:33.959404 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:33.959865 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:33.959905 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.959942 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:33.959973 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.964514 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:34675
I20250626 02:02:33.964589 23142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:34675 every 8 connection(s)
I20250626 02:02:33.965386 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:33.967228 23149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.967237 23147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:33.967339 23146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:33.967563 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:33.967737 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:33.967779 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:33.967804 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903353967803 us; error 0 us; skew 500 ppm
I20250626 02:02:33.969045 20338 webserver.cc:469] Webserver started at http://127.19.220.132:34867/ using document root <none> and password file <none>
I20250626 02:02:33.969216 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:33.969283 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:33.969367 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:33.969442 23143 heartbeater.cc:344] Connected to a master server at 127.19.220.190:41307
I20250626 02:02:33.969551 23143 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:33.969774 23143 heartbeater.cc:507] Master 127.19.220.190:41307 requested a full tablet report, sending...
I20250626 02:02:33.969805 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-3-root/instance:
uuid: "0c3af673b7d14da4a72252188efe1b2a"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.970360 22863 ts_manager.cc:194] Registered new tserver with Master: f3eb450a02d44ab78801e13fb092fef6 (127.19.220.131:34675)
I20250626 02:02:33.970876 22863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39888
I20250626 02:02:33.971405 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20250626 02:02:33.972400 23154 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.972630 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:33.972709 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-3-root
uuid: "0c3af673b7d14da4a72252188efe1b2a"
format_stamp: "Formatted at 2025-06-26 02:02:33 on dist-test-slave-h18f"
I20250626 02:02:33.972779 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750903288526542-20338-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:33.995800 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:33.996301 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:33.996614 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:33.997110 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:33.997169 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:33.997227 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:33.997262 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:34.001662 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.132:43583
I20250626 02:02:34.001824 23217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.132:43583 every 8 connection(s)
I20250626 02:02:34.006023 23218 heartbeater.cc:344] Connected to a master server at 127.19.220.190:41307
I20250626 02:02:34.006122 23218 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:34.006343 23218 heartbeater.cc:507] Master 127.19.220.190:41307 requested a full tablet report, sending...
I20250626 02:02:34.006947 22863 ts_manager.cc:194] Registered new tserver with Master: 0c3af673b7d14da4a72252188efe1b2a (127.19.220.132:43583)
I20250626 02:02:34.007437 22863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39894
I20250626 02:02:34.007716 20338 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.00557317s
I20250626 02:02:34.918273 22992 heartbeater.cc:499] Master 127.19.220.190:41307 was elected leader, sending a full tablet report...
I20250626 02:02:34.946338 23068 heartbeater.cc:499] Master 127.19.220.190:41307 was elected leader, sending a full tablet report...
I20250626 02:02:34.971832 23143 heartbeater.cc:499] Master 127.19.220.190:41307 was elected leader, sending a full tablet report...
I20250626 02:02:35.008306 23218 heartbeater.cc:499] Master 127.19.220.190:41307 was elected leader, sending a full tablet report...
I20250626 02:02:35.038699 20338 test_util.cc:276] Using random seed: -192717847
I20250626 02:02:35.046000 22863 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39914:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 02:02:35.055406 23107 tablet_service.cc:1468] Processing CreateTablet for tablet dd8a2f6871b8431887ccca2b99229ff0 (DEFAULT_TABLE table=test-workload [id=0035348d9ebd4ac78559f7df802810d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:02:35.055744 23107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8a2f6871b8431887ccca2b99229ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:35.056382 23182 tablet_service.cc:1468] Processing CreateTablet for tablet dd8a2f6871b8431887ccca2b99229ff0 (DEFAULT_TABLE table=test-workload [id=0035348d9ebd4ac78559f7df802810d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:02:35.056653 23182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8a2f6871b8431887ccca2b99229ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:35.056674 23032 tablet_service.cc:1468] Processing CreateTablet for tablet dd8a2f6871b8431887ccca2b99229ff0 (DEFAULT_TABLE table=test-workload [id=0035348d9ebd4ac78559f7df802810d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:02:35.056939 23032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8a2f6871b8431887ccca2b99229ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:35.058346 23239 tablet_bootstrap.cc:492] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6: Bootstrap starting.
I20250626 02:02:35.059087 23241 tablet_bootstrap.cc:492] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698: Bootstrap starting.
I20250626 02:02:35.059320 23239 tablet_bootstrap.cc:654] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:35.059988 23240 tablet_bootstrap.cc:492] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a: Bootstrap starting.
I20250626 02:02:35.060194 23241 tablet_bootstrap.cc:654] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:35.060544 23239 tablet_bootstrap.cc:492] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6: No bootstrap required, opened a new log
I20250626 02:02:35.060616 23239 ts_tablet_manager.cc:1397] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:02:35.060994 23239 raft_consensus.cc:357] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.061084 23239 raft_consensus.cc:383] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:35.061105 23239 raft_consensus.cc:738] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3eb450a02d44ab78801e13fb092fef6, State: Initialized, Role: FOLLOWER
I20250626 02:02:35.061220 23241 tablet_bootstrap.cc:492] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698: No bootstrap required, opened a new log
I20250626 02:02:35.061226 23239 consensus_queue.cc:260] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.061321 23241 ts_tablet_manager.cc:1397] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:02:35.061673 23239 ts_tablet_manager.cc:1428] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:35.061798 23241 raft_consensus.cc:357] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.061895 23241 raft_consensus.cc:383] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:35.061949 23241 raft_consensus.cc:738] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fbfc66d7af4defb8f6281a55d9f698, State: Initialized, Role: FOLLOWER
I20250626 02:02:35.062063 23240 tablet_bootstrap.cc:654] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:35.062078 23241 consensus_queue.cc:260] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.062469 23241 ts_tablet_manager.cc:1428] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:35.063185 23240 tablet_bootstrap.cc:492] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a: No bootstrap required, opened a new log
I20250626 02:02:35.063279 23240 ts_tablet_manager.cc:1397] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:02:35.063720 23240 raft_consensus.cc:357] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.063822 23240 raft_consensus.cc:383] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:35.063863 23240 raft_consensus.cc:738] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3af673b7d14da4a72252188efe1b2a, State: Initialized, Role: FOLLOWER
I20250626 02:02:35.063974 23240 consensus_queue.cc:260] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.064339 23240 ts_tablet_manager.cc:1428] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:35.098457 23244 raft_consensus.cc:491] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:02:35.098599 23244 raft_consensus.cc:513] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.099143 23244 leader_election.cc:290] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2fbfc66d7af4defb8f6281a55d9f698 (127.19.220.130:36929), 0c3af673b7d14da4a72252188efe1b2a (127.19.220.132:43583)
I20250626 02:02:35.103507 23042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd8a2f6871b8431887ccca2b99229ff0" candidate_uuid: "f3eb450a02d44ab78801e13fb092fef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" is_pre_election: true
I20250626 02:02:35.103593 23192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd8a2f6871b8431887ccca2b99229ff0" candidate_uuid: "f3eb450a02d44ab78801e13fb092fef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3af673b7d14da4a72252188efe1b2a" is_pre_election: true
I20250626 02:02:35.103675 23042 raft_consensus.cc:2466] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3eb450a02d44ab78801e13fb092fef6 in term 0.
I20250626 02:02:35.103716 23192 raft_consensus.cc:2466] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3eb450a02d44ab78801e13fb092fef6 in term 0.
I20250626 02:02:35.104017 23082 leader_election.cc:304] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c3af673b7d14da4a72252188efe1b2a, f3eb450a02d44ab78801e13fb092fef6; no voters:
I20250626 02:02:35.104207 23244 raft_consensus.cc:2802] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:02:35.104276 23244 raft_consensus.cc:491] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:02:35.104311 23244 raft_consensus.cc:3058] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:35.105149 23244 raft_consensus.cc:513] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.105468 23244 leader_election.cc:290] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [CANDIDATE]: Term 1 election: Requested vote from peers b2fbfc66d7af4defb8f6281a55d9f698 (127.19.220.130:36929), 0c3af673b7d14da4a72252188efe1b2a (127.19.220.132:43583)
I20250626 02:02:35.105810 23192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd8a2f6871b8431887ccca2b99229ff0" candidate_uuid: "f3eb450a02d44ab78801e13fb092fef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3af673b7d14da4a72252188efe1b2a"
I20250626 02:02:35.105810 23042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd8a2f6871b8431887ccca2b99229ff0" candidate_uuid: "f3eb450a02d44ab78801e13fb092fef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2fbfc66d7af4defb8f6281a55d9f698"
I20250626 02:02:35.105926 23192 raft_consensus.cc:3058] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:35.105990 23042 raft_consensus.cc:3058] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:35.106819 23192 raft_consensus.cc:2466] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3eb450a02d44ab78801e13fb092fef6 in term 1.
I20250626 02:02:35.106832 23042 raft_consensus.cc:2466] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3eb450a02d44ab78801e13fb092fef6 in term 1.
I20250626 02:02:35.107079 23082 leader_election.cc:304] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c3af673b7d14da4a72252188efe1b2a, f3eb450a02d44ab78801e13fb092fef6; no voters:
I20250626 02:02:35.107249 23244 raft_consensus.cc:2802] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:02:35.107470 23244 raft_consensus.cc:695] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 1 LEADER]: Becoming Leader. State: Replica: f3eb450a02d44ab78801e13fb092fef6, State: Running, Role: LEADER
I20250626 02:02:35.107617 23244 consensus_queue.cc:237] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } }
I20250626 02:02:35.109109 22862 catalog_manager.cc:5582] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3eb450a02d44ab78801e13fb092fef6 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "f3eb450a02d44ab78801e13fb092fef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3eb450a02d44ab78801e13fb092fef6" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 34675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 } health_report { overall_health: UNKNOWN } } }
W20250626 02:02:35.190260 23069 tablet.cc:2378] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 02:02:35.514534 23244 consensus_queue.cc:1035] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2fbfc66d7af4defb8f6281a55d9f698" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:02:35.517740 23244 consensus_queue.cc:1035] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3af673b7d14da4a72252188efe1b2a" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:02:35.819031 22917 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:35.819324 22917 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:35.819404 22917 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:37.819720 22917 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:37.820021 22917 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:37.820086 22917 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:39.820391 22917 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:39.820729 22917 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:39.820791 22917 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:41.821072 22917 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:41.821409 22917 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:41.821477 22917 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:43.821780 22917 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:43.822119 22917 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:43.822189 22917 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:45.822502 22917 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:45.822867 22917 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:45.822942 22917 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:46.174193 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:02:46.177860 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:46.190821 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:02:46.192575 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:02:46.196301 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:46.196494 20338 tablet_replica.cc:331] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698: stopping tablet replica
I20250626 02:02:46.196630 20338 raft_consensus.cc:2241] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:02:46.196758 20338 raft_consensus.cc:2270] T dd8a2f6871b8431887ccca2b99229ff0 P b2fbfc66d7af4defb8f6281a55d9f698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:46.209285 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:02:46.211236 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:02:46.215135 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:46.215307 20338 tablet_replica.cc:331] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6: stopping tablet replica
I20250626 02:02:46.215416 20338 raft_consensus.cc:2241] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:02:46.215615 20338 raft_consensus.cc:2270] T dd8a2f6871b8431887ccca2b99229ff0 P f3eb450a02d44ab78801e13fb092fef6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:46.228308 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:02:46.230310 20338 tablet_server.cc:178] TabletServer@127.19.220.132:0 shutting down...
I20250626 02:02:46.234653 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:46.234829 20338 tablet_replica.cc:331] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a: stopping tablet replica
I20250626 02:02:46.234941 20338 raft_consensus.cc:2241] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:02:46.235044 20338 raft_consensus.cc:2270] T dd8a2f6871b8431887ccca2b99229ff0 P 0c3af673b7d14da4a72252188efe1b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:46.247541 20338 tablet_server.cc:195] TabletServer@127.19.220.132:0 shutdown complete.
I20250626 02:02:46.249452 20338 master.cc:561] Master@127.19.220.190:41307 shutting down...
I20250626 02:02:46.253804 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:02:46.253975 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:46.254055 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6536e132c2e8422aa0a153dd45bb8188: stopping tablet replica
I20250626 02:02:46.266423 20338 master.cc:583] Master@127.19.220.190:41307 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12483 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20250626 02:02:46.272938 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:33423
I20250626 02:02:46.273259 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:46.275231 23263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:46.275266 23265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:46.275393 20338 server_base.cc:1048] running on GCE node
W20250626 02:02:46.275394 23262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:46.275648 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:46.275718 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:46.275754 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903366275753 us; error 0 us; skew 500 ppm
I20250626 02:02:46.282424 20338 webserver.cc:469] Webserver started at http://127.19.220.190:43315/ using document root <none> and password file <none>
I20250626 02:02:46.282599 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:46.282668 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:46.282765 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:46.283152 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "117f27d9560f4a979bdf4d7dc94c9fae"
format_stamp: "Formatted at 2025-06-26 02:02:46 on dist-test-slave-h18f"
I20250626 02:02:46.284683 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:02:46.285665 23270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.285892 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:46.285956 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "117f27d9560f4a979bdf4d7dc94c9fae"
format_stamp: "Formatted at 2025-06-26 02:02:46 on dist-test-slave-h18f"
I20250626 02:02:46.286024 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:46.304461 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:46.304857 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:46.308687 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:33423
I20250626 02:02:46.310748 23322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:33423 every 8 connection(s)
I20250626 02:02:46.311200 23323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:46.313097 23323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae: Bootstrap starting.
I20250626 02:02:46.313944 23323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:46.314924 23323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae: No bootstrap required, opened a new log
I20250626 02:02:46.315282 23323 raft_consensus.cc:357] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f27d9560f4a979bdf4d7dc94c9fae" member_type: VOTER }
I20250626 02:02:46.315368 23323 raft_consensus.cc:383] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:46.315405 23323 raft_consensus.cc:738] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117f27d9560f4a979bdf4d7dc94c9fae, State: Initialized, Role: FOLLOWER
I20250626 02:02:46.315572 23323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f27d9560f4a979bdf4d7dc94c9fae" member_type: VOTER }
I20250626 02:02:46.315645 23323 raft_consensus.cc:397] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:02:46.315688 23323 raft_consensus.cc:491] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:02:46.315742 23323 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:46.316484 23323 raft_consensus.cc:513] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f27d9560f4a979bdf4d7dc94c9fae" member_type: VOTER }
I20250626 02:02:46.316612 23323 leader_election.cc:304] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 117f27d9560f4a979bdf4d7dc94c9fae; no voters:
I20250626 02:02:46.316777 23323 leader_election.cc:290] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:02:46.316886 23326 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:02:46.317096 23326 raft_consensus.cc:695] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 1 LEADER]: Becoming Leader. State: Replica: 117f27d9560f4a979bdf4d7dc94c9fae, State: Running, Role: LEADER
I20250626 02:02:46.317200 23323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:02:46.317219 23326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f27d9560f4a979bdf4d7dc94c9fae" member_type: VOTER }
I20250626 02:02:46.317641 23327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "117f27d9560f4a979bdf4d7dc94c9fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f27d9560f4a979bdf4d7dc94c9fae" member_type: VOTER } }
I20250626 02:02:46.317693 23328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 117f27d9560f4a979bdf4d7dc94c9fae. Latest consensus state: current_term: 1 leader_uuid: "117f27d9560f4a979bdf4d7dc94c9fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f27d9560f4a979bdf4d7dc94c9fae" member_type: VOTER } }
I20250626 02:02:46.317829 23328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:46.317813 23327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:46.318359 23332 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:02:46.319082 23332 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:02:46.319259 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:02:46.321970 23332 catalog_manager.cc:1349] Generated new cluster ID: 3ad7c8be219b47a1addae20faae270d8
I20250626 02:02:46.322037 23332 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:02:46.347364 23332 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:02:46.347923 23332 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:02:46.354400 23332 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae: Generated new TSK 0
I20250626 02:02:46.354575 23332 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:02:46.384027 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:46.386015 23344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:46.386082 23347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:46.386224 20338 server_base.cc:1048] running on GCE node
W20250626 02:02:46.386015 23345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:46.386492 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:46.386539 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:46.386575 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903366386575 us; error 0 us; skew 500 ppm
I20250626 02:02:46.387540 20338 webserver.cc:469] Webserver started at http://127.19.220.129:46843/ using document root <none> and password file <none>
I20250626 02:02:46.387696 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:46.387744 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:46.387830 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:46.388298 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "4874266ce8aa4c06826d3fe9ef003b6e"
format_stamp: "Formatted at 2025-06-26 02:02:46 on dist-test-slave-h18f"
I20250626 02:02:46.389811 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:46.390751 23352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.391048 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:46.391112 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "4874266ce8aa4c06826d3fe9ef003b6e"
format_stamp: "Formatted at 2025-06-26 02:02:46 on dist-test-slave-h18f"
I20250626 02:02:46.391166 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:46.412997 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:46.413395 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:46.413657 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:46.414180 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:46.414222 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.414258 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:46.414289 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.418613 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:43019
I20250626 02:02:46.418637 23415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:43019 every 8 connection(s)
I20250626 02:02:46.419549 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:46.421425 23420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:46.421564 20338 server_base.cc:1048] running on GCE node
W20250626 02:02:46.421367 23421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:46.421443 23423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:46.421952 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:46.422010 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:46.422032 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903366422032 us; error 0 us; skew 500 ppm
I20250626 02:02:46.423043 20338 webserver.cc:469] Webserver started at http://127.19.220.130:38333/ using document root <none> and password file <none>
I20250626 02:02:46.423195 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:46.423251 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:46.423328 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:46.423782 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "f41bb356ddd2400fa2e95d708f7f4ab1"
format_stamp: "Formatted at 2025-06-26 02:02:46 on dist-test-slave-h18f"
I20250626 02:02:46.423897 23416 heartbeater.cc:344] Connected to a master server at 127.19.220.190:33423
I20250626 02:02:46.423974 23416 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:46.424185 23416 heartbeater.cc:507] Master 127.19.220.190:33423 requested a full tablet report, sending...
I20250626 02:02:46.424796 23287 ts_manager.cc:194] Registered new tserver with Master: 4874266ce8aa4c06826d3fe9ef003b6e (127.19.220.129:43019)
I20250626 02:02:46.425405 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:02:46.425657 23287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36252
I20250626 02:02:46.426348 23428 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.426549 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:02:46.426610 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "f41bb356ddd2400fa2e95d708f7f4ab1"
format_stamp: "Formatted at 2025-06-26 02:02:46 on dist-test-slave-h18f"
I20250626 02:02:46.426676 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:46.436605 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:46.437036 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:46.437323 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:46.437788 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:46.437826 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.437878 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:46.437911 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.441967 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:41863
I20250626 02:02:46.442025 23491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:41863 every 8 connection(s)
I20250626 02:02:46.442750 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:46.444577 23495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:46.444605 23496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:46.444798 23498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:46.444936 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:46.445106 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:46.445161 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:46.445184 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903366445184 us; error 0 us; skew 500 ppm
I20250626 02:02:46.446244 20338 webserver.cc:469] Webserver started at http://127.19.220.131:35245/ using document root <none> and password file <none>
I20250626 02:02:46.446395 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:46.446457 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:46.446537 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:46.447019 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "64de3a5d4bee438683c977891d47341f"
format_stamp: "Formatted at 2025-06-26 02:02:46 on dist-test-slave-h18f"
I20250626 02:02:46.447563 23492 heartbeater.cc:344] Connected to a master server at 127.19.220.190:33423
I20250626 02:02:46.447670 23492 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:46.447921 23492 heartbeater.cc:507] Master 127.19.220.190:33423 requested a full tablet report, sending...
I20250626 02:02:46.448601 23287 ts_manager.cc:194] Registered new tserver with Master: f41bb356ddd2400fa2e95d708f7f4ab1 (127.19.220.130:41863)
I20250626 02:02:46.448844 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 02:02:46.449133 23287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36258
I20250626 02:02:46.449888 23503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.450176 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:46.450241 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "64de3a5d4bee438683c977891d47341f"
format_stamp: "Formatted at 2025-06-26 02:02:46 on dist-test-slave-h18f"
I20250626 02:02:46.450314 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:46.469496 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:46.469956 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:46.470260 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:46.470731 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:46.470772 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.470820 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:46.470856 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:46.475322 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:32781
I20250626 02:02:46.475379 23566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:32781 every 8 connection(s)
I20250626 02:02:46.480022 23567 heartbeater.cc:344] Connected to a master server at 127.19.220.190:33423
I20250626 02:02:46.480152 23567 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:46.480389 23567 heartbeater.cc:507] Master 127.19.220.190:33423 requested a full tablet report, sending...
I20250626 02:02:46.480981 23287 ts_manager.cc:194] Registered new tserver with Master: 64de3a5d4bee438683c977891d47341f (127.19.220.131:32781)
I20250626 02:02:46.481292 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005597481s
I20250626 02:02:46.481585 23287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36268
I20250626 02:02:47.426563 23416 heartbeater.cc:499] Master 127.19.220.190:33423 was elected leader, sending a full tablet report...
I20250626 02:02:47.450062 23492 heartbeater.cc:499] Master 127.19.220.190:33423 was elected leader, sending a full tablet report...
I20250626 02:02:47.483671 23567 heartbeater.cc:499] Master 127.19.220.190:33423 was elected leader, sending a full tablet report...
I20250626 02:02:47.512214 20338 test_util.cc:276] Using random seed: -180244332
I20250626 02:02:47.519781 23287 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36280:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:02:47.520320 23287 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:02:47.529649 23531 tablet_service.cc:1468] Processing CreateTablet for tablet cbff16cbb02642aebef6ab39b485200f (DEFAULT_TABLE table=test-workload [id=b6450ebba3a7443d8c1987d67de95ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:02:47.529721 23380 tablet_service.cc:1468] Processing CreateTablet for tablet cbff16cbb02642aebef6ab39b485200f (DEFAULT_TABLE table=test-workload [id=b6450ebba3a7443d8c1987d67de95ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:02:47.530004 23531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbff16cbb02642aebef6ab39b485200f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:47.530004 23380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbff16cbb02642aebef6ab39b485200f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:47.532575 23588 tablet_bootstrap.cc:492] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e: Bootstrap starting.
I20250626 02:02:47.532586 23456 tablet_service.cc:1468] Processing CreateTablet for tablet cbff16cbb02642aebef6ab39b485200f (DEFAULT_TABLE table=test-workload [id=b6450ebba3a7443d8c1987d67de95ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:02:47.532842 23589 tablet_bootstrap.cc:492] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f: Bootstrap starting.
I20250626 02:02:47.532898 23456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbff16cbb02642aebef6ab39b485200f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:47.533708 23589 tablet_bootstrap.cc:654] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:47.533759 23588 tablet_bootstrap.cc:654] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:47.534802 23589 tablet_bootstrap.cc:492] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f: No bootstrap required, opened a new log
I20250626 02:02:47.534868 23589 ts_tablet_manager.cc:1397] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:02:47.534883 23588 tablet_bootstrap.cc:492] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e: No bootstrap required, opened a new log
I20250626 02:02:47.534965 23588 ts_tablet_manager.cc:1397] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:02:47.535212 23589 raft_consensus.cc:357] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.535317 23589 raft_consensus.cc:383] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:47.535341 23592 tablet_bootstrap.cc:492] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1: Bootstrap starting.
I20250626 02:02:47.535353 23589 raft_consensus.cc:738] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64de3a5d4bee438683c977891d47341f, State: Initialized, Role: FOLLOWER
I20250626 02:02:47.535387 23588 raft_consensus.cc:357] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.535521 23588 raft_consensus.cc:383] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:47.535562 23588 raft_consensus.cc:738] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4874266ce8aa4c06826d3fe9ef003b6e, State: Initialized, Role: FOLLOWER
I20250626 02:02:47.535523 23589 consensus_queue.cc:260] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.535674 23588 consensus_queue.cc:260] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.535920 23589 ts_tablet_manager.cc:1428] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:47.536015 23588 ts_tablet_manager.cc:1428] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:47.536490 23592 tablet_bootstrap.cc:654] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:47.537564 23592 tablet_bootstrap.cc:492] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1: No bootstrap required, opened a new log
I20250626 02:02:47.537714 23592 ts_tablet_manager.cc:1397] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:02:47.538208 23592 raft_consensus.cc:357] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.538352 23592 raft_consensus.cc:383] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:47.538419 23592 raft_consensus.cc:738] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41bb356ddd2400fa2e95d708f7f4ab1, State: Initialized, Role: FOLLOWER
I20250626 02:02:47.538563 23592 consensus_queue.cc:260] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.538930 23592 ts_tablet_manager.cc:1428] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:47.641676 23594 raft_consensus.cc:491] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:02:47.641840 23594 raft_consensus.cc:513] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.642416 23594 leader_election.cc:290] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64de3a5d4bee438683c977891d47341f (127.19.220.131:32781), f41bb356ddd2400fa2e95d708f7f4ab1 (127.19.220.130:41863)
I20250626 02:02:47.646656 23541 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbff16cbb02642aebef6ab39b485200f" candidate_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de3a5d4bee438683c977891d47341f" is_pre_election: true
I20250626 02:02:47.646658 23466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbff16cbb02642aebef6ab39b485200f" candidate_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" is_pre_election: true
I20250626 02:02:47.646831 23541 raft_consensus.cc:2466] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4874266ce8aa4c06826d3fe9ef003b6e in term 0.
I20250626 02:02:47.646878 23466 raft_consensus.cc:2466] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4874266ce8aa4c06826d3fe9ef003b6e in term 0.
I20250626 02:02:47.647136 23356 leader_election.cc:304] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4874266ce8aa4c06826d3fe9ef003b6e, 64de3a5d4bee438683c977891d47341f; no voters:
I20250626 02:02:47.647306 23594 raft_consensus.cc:2802] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:02:47.647372 23594 raft_consensus.cc:491] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:02:47.647419 23594 raft_consensus.cc:3058] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:47.648325 23594 raft_consensus.cc:513] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.648664 23594 leader_election.cc:290] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [CANDIDATE]: Term 1 election: Requested vote from peers 64de3a5d4bee438683c977891d47341f (127.19.220.131:32781), f41bb356ddd2400fa2e95d708f7f4ab1 (127.19.220.130:41863)
I20250626 02:02:47.648923 23541 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbff16cbb02642aebef6ab39b485200f" candidate_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de3a5d4bee438683c977891d47341f"
I20250626 02:02:47.648967 23466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbff16cbb02642aebef6ab39b485200f" candidate_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1"
I20250626 02:02:47.649022 23541 raft_consensus.cc:3058] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:47.649055 23466 raft_consensus.cc:3058] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:47.649891 23541 raft_consensus.cc:2466] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4874266ce8aa4c06826d3fe9ef003b6e in term 1.
I20250626 02:02:47.649952 23466 raft_consensus.cc:2466] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4874266ce8aa4c06826d3fe9ef003b6e in term 1.
I20250626 02:02:47.650224 23355 leader_election.cc:304] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4874266ce8aa4c06826d3fe9ef003b6e, f41bb356ddd2400fa2e95d708f7f4ab1; no voters:
I20250626 02:02:47.650382 23594 raft_consensus.cc:2802] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:02:47.650609 23594 raft_consensus.cc:695] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 1 LEADER]: Becoming Leader. State: Replica: 4874266ce8aa4c06826d3fe9ef003b6e, State: Running, Role: LEADER
I20250626 02:02:47.650787 23594 consensus_queue.cc:237] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } }
I20250626 02:02:47.652231 23285 catalog_manager.cc:5582] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4874266ce8aa4c06826d3fe9ef003b6e (127.19.220.129). New cstate: current_term: 1 leader_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4874266ce8aa4c06826d3fe9ef003b6e" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 43019 } health_report { overall_health: HEALTHY } } }
I20250626 02:02:48.070672 23594 consensus_queue.cc:1035] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41bb356ddd2400fa2e95d708f7f4ab1" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:02:48.073768 23594 consensus_queue.cc:1035] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64de3a5d4bee438683c977891d47341f" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:02:48.319195 23341 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:48.319559 23341 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:48.319641 23341 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:48.320006 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:49.321775 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:50.319939 23341 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:50.320273 23341 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:50.320330 23341 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:50.323124 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:51.324584 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:52.320611 23341 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:52.320917 23341 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:52.320971 23341 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:52.326166 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:53.327730 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:54.321249 23341 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:54.321553 23341 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:54.321604 23341 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:54.329217 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:55.330725 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:56.321861 23341 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:56.322139 23341 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:56.322187 23341 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:56.332155 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:57.333590 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:58.322526 23341 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:02:58.322839 23341 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:02:58.322898 23341 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:02:58.335070 23336 placement_policy_util.cc:407] tablet cbff16cbb02642aebef6ab39b485200f: detected majority of replicas (2 of 3) at location L0
I20250626 02:02:58.727308 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:02:58.730865 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:58.731076 20338 tablet_replica.cc:331] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e: stopping tablet replica
I20250626 02:02:58.731215 20338 raft_consensus.cc:2241] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:02:58.731411 20338 raft_consensus.cc:2270] T cbff16cbb02642aebef6ab39b485200f P 4874266ce8aa4c06826d3fe9ef003b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:58.744091 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:02:58.746024 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:02:58.749166 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:58.749341 20338 tablet_replica.cc:331] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1: stopping tablet replica
I20250626 02:02:58.749497 20338 raft_consensus.cc:2241] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:02:58.749608 20338 raft_consensus.cc:2270] T cbff16cbb02642aebef6ab39b485200f P f41bb356ddd2400fa2e95d708f7f4ab1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:58.761919 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:02:58.763629 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:02:58.767277 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:02:58.767437 20338 tablet_replica.cc:331] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f: stopping tablet replica
I20250626 02:02:58.767552 20338 raft_consensus.cc:2241] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:02:58.767663 20338 raft_consensus.cc:2270] T cbff16cbb02642aebef6ab39b485200f P 64de3a5d4bee438683c977891d47341f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:58.780205 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:02:58.782007 20338 master.cc:561] Master@127.19.220.190:33423 shutting down...
I20250626 02:02:58.784930 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:02:58.785058 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:02:58.785106 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P 117f27d9560f4a979bdf4d7dc94c9fae: stopping tablet replica
I20250626 02:02:58.797231 20338 master.cc:583] Master@127.19.220.190:33423 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12530 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20250626 02:02:58.803671 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:42171
I20250626 02:02:58.803969 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:58.805624 23615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:58.805716 20338 server_base.cc:1048] running on GCE node
W20250626 02:02:58.805786 23614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:58.805830 23617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:58.806131 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:58.806170 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:58.806185 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903378806185 us; error 0 us; skew 500 ppm
I20250626 02:02:58.811964 20338 webserver.cc:469] Webserver started at http://127.19.220.190:33271/ using document root <none> and password file <none>
I20250626 02:02:58.812152 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:58.812218 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:58.812302 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:58.812722 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "37bae5288e804018b4dc9a3f7533d06c"
format_stamp: "Formatted at 2025-06-26 02:02:58 on dist-test-slave-h18f"
I20250626 02:02:58.814160 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:02:58.815028 23622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:58.815292 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:58.815351 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "37bae5288e804018b4dc9a3f7533d06c"
format_stamp: "Formatted at 2025-06-26 02:02:58 on dist-test-slave-h18f"
I20250626 02:02:58.815407 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:58.845527 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:58.845894 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:58.850021 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:42171
I20250626 02:02:58.851969 23674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:42171 every 8 connection(s)
I20250626 02:02:58.852452 23675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:02:58.854246 23675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c: Bootstrap starting.
I20250626 02:02:58.854991 23675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c: Neither blocks nor log segments found. Creating new log.
I20250626 02:02:58.855893 23675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c: No bootstrap required, opened a new log
I20250626 02:02:58.856292 23675 raft_consensus.cc:357] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bae5288e804018b4dc9a3f7533d06c" member_type: VOTER }
I20250626 02:02:58.856374 23675 raft_consensus.cc:383] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:02:58.856396 23675 raft_consensus.cc:738] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bae5288e804018b4dc9a3f7533d06c, State: Initialized, Role: FOLLOWER
I20250626 02:02:58.856523 23675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bae5288e804018b4dc9a3f7533d06c" member_type: VOTER }
I20250626 02:02:58.856593 23675 raft_consensus.cc:397] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:02:58.856618 23675 raft_consensus.cc:491] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:02:58.856674 23675 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:02:58.857326 23675 raft_consensus.cc:513] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bae5288e804018b4dc9a3f7533d06c" member_type: VOTER }
I20250626 02:02:58.857430 23675 leader_election.cc:304] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37bae5288e804018b4dc9a3f7533d06c; no voters:
I20250626 02:02:58.857614 23675 leader_election.cc:290] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:02:58.857753 23678 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:02:58.857967 23678 raft_consensus.cc:695] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 1 LEADER]: Becoming Leader. State: Replica: 37bae5288e804018b4dc9a3f7533d06c, State: Running, Role: LEADER
I20250626 02:02:58.858095 23675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:02:58.858091 23678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bae5288e804018b4dc9a3f7533d06c" member_type: VOTER }
I20250626 02:02:58.858513 23679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37bae5288e804018b4dc9a3f7533d06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bae5288e804018b4dc9a3f7533d06c" member_type: VOTER } }
I20250626 02:02:58.858620 23679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:58.858527 23680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37bae5288e804018b4dc9a3f7533d06c. Latest consensus state: current_term: 1 leader_uuid: "37bae5288e804018b4dc9a3f7533d06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37bae5288e804018b4dc9a3f7533d06c" member_type: VOTER } }
I20250626 02:02:58.858763 23680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [sys.catalog]: This master's current role is: LEADER
I20250626 02:02:58.859162 23683 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:02:58.859923 23683 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:02:58.860109 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:02:58.861698 23683 catalog_manager.cc:1349] Generated new cluster ID: b07fe7720efb4168ab7d835dcc6a3d09
I20250626 02:02:58.861758 23683 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:02:58.877496 23683 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:02:58.878026 23683 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:02:58.881629 23683 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c: Generated new TSK 0
I20250626 02:02:58.881778 23683 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:02:58.892758 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:02:58.894852 20338 server_base.cc:1048] running on GCE node
W20250626 02:02:58.894888 23697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:58.894984 23696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:58.895008 23699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:58.895270 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:58.895316 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:58.895332 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903378895332 us; error 0 us; skew 500 ppm
I20250626 02:02:58.896250 20338 webserver.cc:469] Webserver started at http://127.19.220.129:45171/ using document root <none> and password file <none>
I20250626 02:02:58.896399 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:58.896445 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:58.896533 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:58.896914 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "3a8897880da64721ad84d66ff1fcf3a4"
format_stamp: "Formatted at 2025-06-26 02:02:58 on dist-test-slave-h18f"
I20250626 02:02:58.898288 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:02:58.899114 23704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:58.899336 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:58.899399 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "3a8897880da64721ad84d66ff1fcf3a4"
format_stamp: "Formatted at 2025-06-26 02:02:58 on dist-test-slave-h18f"
I20250626 02:02:58.899468 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:58.916110 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:58.916596 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:58.916878 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:58.917304 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:58.917340 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:58.917392 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:58.917424 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:58.921312 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:34725
I20250626 02:02:58.921370 23767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:34725 every 8 connection(s)
I20250626 02:02:58.922103 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:58.924027 23775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:58.924206 23773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:58.924048 23772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:58.924366 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:58.924523 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:58.924566 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:58.924587 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903378924586 us; error 0 us; skew 500 ppm
I20250626 02:02:58.925658 20338 webserver.cc:469] Webserver started at http://127.19.220.130:39613/ using document root <none> and password file <none>
I20250626 02:02:58.925817 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:58.925884 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:58.926002 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:58.926494 23768 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42171
I20250626 02:02:58.926497 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "6c2126b790c0477595d5b94c658b8a80"
format_stamp: "Formatted at 2025-06-26 02:02:58 on dist-test-slave-h18f"
I20250626 02:02:58.926633 23768 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:58.926836 23768 heartbeater.cc:507] Master 127.19.220.190:42171 requested a full tablet report, sending...
I20250626 02:02:58.927471 23639 ts_manager.cc:194] Registered new tserver with Master: 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725)
I20250626 02:02:58.928241 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:58.928407 23639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41592
I20250626 02:02:58.929243 23780 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:58.929462 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:02:58.929524 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "6c2126b790c0477595d5b94c658b8a80"
format_stamp: "Formatted at 2025-06-26 02:02:58 on dist-test-slave-h18f"
I20250626 02:02:58.929592 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:58.937274 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:58.937579 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:58.937860 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:58.938252 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:58.938292 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:58.938336 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:58.938369 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:58.942662 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:39183
I20250626 02:02:58.942724 23843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:39183 every 8 connection(s)
I20250626 02:02:58.943411 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:02:58.945271 23848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:58.945297 23847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:02:58.945502 23850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:02:58.945614 20338 server_base.cc:1048] running on GCE node
I20250626 02:02:58.945807 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:02:58.945866 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:02:58.945888 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903378945888 us; error 0 us; skew 500 ppm
I20250626 02:02:58.946955 20338 webserver.cc:469] Webserver started at http://127.19.220.131:43263/ using document root <none> and password file <none>
I20250626 02:02:58.947105 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:02:58.947155 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:02:58.947228 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:02:58.947315 23844 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42171
I20250626 02:02:58.947399 23844 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:58.947616 23844 heartbeater.cc:507] Master 127.19.220.190:42171 requested a full tablet report, sending...
I20250626 02:02:58.947671 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "d150470bce35446b9b3f41ec78cb5064"
format_stamp: "Formatted at 2025-06-26 02:02:58 on dist-test-slave-h18f"
I20250626 02:02:58.948217 23639 ts_manager.cc:194] Registered new tserver with Master: 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:02:58.948709 23639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41598
I20250626 02:02:58.949178 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:02:58.950064 23855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:58.950279 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:02:58.950337 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "d150470bce35446b9b3f41ec78cb5064"
format_stamp: "Formatted at 2025-06-26 02:02:58 on dist-test-slave-h18f"
I20250626 02:02:58.950407 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:02:58.999138 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:02:58.999609 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:02:58.999918 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:02:59.000437 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:02:59.000478 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:59.000528 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:02:59.000564 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:02:59.004993 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:40133
I20250626 02:02:59.005079 23918 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:40133 every 8 connection(s)
I20250626 02:02:59.009286 23919 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42171
I20250626 02:02:59.009370 23919 heartbeater.cc:461] Registering TS with master...
I20250626 02:02:59.009567 23919 heartbeater.cc:507] Master 127.19.220.190:42171 requested a full tablet report, sending...
I20250626 02:02:59.010159 23639 ts_manager.cc:194] Registered new tserver with Master: d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:02:59.010630 23639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41612
I20250626 02:02:59.011037 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005630486s
I20250626 02:02:59.929301 23768 heartbeater.cc:499] Master 127.19.220.190:42171 was elected leader, sending a full tablet report...
I20250626 02:02:59.949636 23844 heartbeater.cc:499] Master 127.19.220.190:42171 was elected leader, sending a full tablet report...
I20250626 02:03:00.011436 23919 heartbeater.cc:499] Master 127.19.220.190:42171 was elected leader, sending a full tablet report...
I20250626 02:03:00.037078 20338 test_util.cc:276] Using random seed: -167719469
I20250626 02:03:00.045024 23639 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41624:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:03:00.045641 23639 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:03:00.058959 23808 tablet_service.cc:1468] Processing CreateTablet for tablet 363d650f886244d38b78e6af7bb5f6ce (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250626 02:03:00.059319 23808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363d650f886244d38b78e6af7bb5f6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.059487 23807 tablet_service.cc:1468] Processing CreateTablet for tablet 65bf2a25991b4d4a94efbc900569a162 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250626 02:03:00.059721 23807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bf2a25991b4d4a94efbc900569a162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.060246 23732 tablet_service.cc:1468] Processing CreateTablet for tablet 363d650f886244d38b78e6af7bb5f6ce (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250626 02:03:00.060375 23806 tablet_service.cc:1468] Processing CreateTablet for tablet 894851ab3c1e4d79971597e62b99cfbf (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250626 02:03:00.060518 23732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363d650f886244d38b78e6af7bb5f6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.060626 23806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894851ab3c1e4d79971597e62b99cfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.062093 23939 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.062244 23807 tablet_service.cc:1468] Processing CreateTablet for tablet 13501c41bba94a7bbfb48a2715cd2d40 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250626 02:03:00.062475 23807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13501c41bba94a7bbfb48a2715cd2d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.062623 23732 tablet_service.cc:1468] Processing CreateTablet for tablet 13501c41bba94a7bbfb48a2715cd2d40 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250626 02:03:00.062790 23806 tablet_service.cc:1468] Processing CreateTablet for tablet 34ac2721a9bb44efa815122997c53fd7 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250626 02:03:00.062867 23732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13501c41bba94a7bbfb48a2715cd2d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.062988 23806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac2721a9bb44efa815122997c53fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.063298 23939 tablet_bootstrap.cc:654] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.064342 23805 tablet_service.cc:1468] Processing CreateTablet for tablet aa3a5817958042e8b2753a3df8e2ba4c (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250626 02:03:00.064376 23723 tablet_service.cc:1468] Processing CreateTablet for tablet e37650dfb570400fb02742cb1220c7e1 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250626 02:03:00.064569 23805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3a5817958042e8b2753a3df8e2ba4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.064651 23723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37650dfb570400fb02742cb1220c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.065111 23732 tablet_service.cc:1468] Processing CreateTablet for tablet 34ac2721a9bb44efa815122997c53fd7 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250626 02:03:00.065359 23732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac2721a9bb44efa815122997c53fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.066661 23804 tablet_service.cc:1468] Processing CreateTablet for tablet 70698cc805a448b28cef88e47e146427 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250626 02:03:00.066905 23804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70698cc805a448b28cef88e47e146427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.066929 23724 tablet_service.cc:1468] Processing CreateTablet for tablet 093974e120e643bc93fcb64da8f31ee1 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250626 02:03:00.067159 23724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 093974e120e643bc93fcb64da8f31ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.067523 23730 tablet_service.cc:1468] Processing CreateTablet for tablet 894851ab3c1e4d79971597e62b99cfbf (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250626 02:03:00.067751 23730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894851ab3c1e4d79971597e62b99cfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.069005 23803 tablet_service.cc:1468] Processing CreateTablet for tablet e38ec4282a4b40e8bb9d07ef486659d6 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250626 02:03:00.069271 23803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38ec4282a4b40e8bb9d07ef486659d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.069525 23883 tablet_service.cc:1468] Processing CreateTablet for tablet 363d650f886244d38b78e6af7bb5f6ce (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250626 02:03:00.069739 23883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363d650f886244d38b78e6af7bb5f6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.069880 23725 tablet_service.cc:1468] Processing CreateTablet for tablet dbd6b437781b4876af28add914954628 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250626 02:03:00.070125 23725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbd6b437781b4876af28add914954628. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.070202 23729 tablet_service.cc:1468] Processing CreateTablet for tablet aa3a5817958042e8b2753a3df8e2ba4c (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250626 02:03:00.070436 23729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3a5817958042e8b2753a3df8e2ba4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.071465 23802 tablet_service.cc:1468] Processing CreateTablet for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250626 02:03:00.071693 23802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.071978 23883 tablet_service.cc:1468] Processing CreateTablet for tablet 13501c41bba94a7bbfb48a2715cd2d40 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250626 02:03:00.072346 23882 tablet_service.cc:1468] Processing CreateTablet for tablet 65bf2a25991b4d4a94efbc900569a162 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250626 02:03:00.072499 23728 tablet_service.cc:1468] Processing CreateTablet for tablet 70698cc805a448b28cef88e47e146427 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250626 02:03:00.072589 23882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bf2a25991b4d4a94efbc900569a162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.072757 23728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70698cc805a448b28cef88e47e146427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.073007 23726 tablet_service.cc:1468] Processing CreateTablet for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250626 02:03:00.073230 23726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.073763 23801 tablet_service.cc:1468] Processing CreateTablet for tablet dbd6b437781b4876af28add914954628 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250626 02:03:00.073982 23801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbd6b437781b4876af28add914954628. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.074568 23882 tablet_service.cc:1468] Processing CreateTablet for tablet 34ac2721a9bb44efa815122997c53fd7 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250626 02:03:00.074777 23882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac2721a9bb44efa815122997c53fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.074971 23727 tablet_service.cc:1468] Processing CreateTablet for tablet e38ec4282a4b40e8bb9d07ef486659d6 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250626 02:03:00.064378 23731 tablet_service.cc:1468] Processing CreateTablet for tablet 65bf2a25991b4d4a94efbc900569a162 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250626 02:03:00.075198 23727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38ec4282a4b40e8bb9d07ef486659d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.075851 23874 tablet_service.cc:1468] Processing CreateTablet for tablet e37650dfb570400fb02742cb1220c7e1 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250626 02:03:00.075980 23800 tablet_service.cc:1468] Processing CreateTablet for tablet 093974e120e643bc93fcb64da8f31ee1 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250626 02:03:00.076088 23874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37650dfb570400fb02742cb1220c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.076220 23731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bf2a25991b4d4a94efbc900569a162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.076242 23800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 093974e120e643bc93fcb64da8f31ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.077869 23799 tablet_service.cc:1468] Processing CreateTablet for tablet e37650dfb570400fb02742cb1220c7e1 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250626 02:03:00.078045 23799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37650dfb570400fb02742cb1220c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.078249 23875 tablet_service.cc:1468] Processing CreateTablet for tablet 093974e120e643bc93fcb64da8f31ee1 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250626 02:03:00.078500 23875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 093974e120e643bc93fcb64da8f31ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.079564 23881 tablet_service.cc:1468] Processing CreateTablet for tablet 894851ab3c1e4d79971597e62b99cfbf (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250626 02:03:00.079810 23881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894851ab3c1e4d79971597e62b99cfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.080678 23876 tablet_service.cc:1468] Processing CreateTablet for tablet dbd6b437781b4876af28add914954628 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250626 02:03:00.080936 23876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbd6b437781b4876af28add914954628. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.081918 23940 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.082136 23942 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.082443 23880 tablet_service.cc:1468] Processing CreateTablet for tablet aa3a5817958042e8b2753a3df8e2ba4c (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250626 02:03:00.082702 23880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3a5817958042e8b2753a3df8e2ba4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.083110 23877 tablet_service.cc:1468] Processing CreateTablet for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250626 02:03:00.083340 23877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.083570 23878 tablet_service.cc:1468] Processing CreateTablet for tablet e38ec4282a4b40e8bb9d07ef486659d6 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250626 02:03:00.083830 23878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38ec4282a4b40e8bb9d07ef486659d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.083310 23940 tablet_bootstrap.cc:654] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.085050 23939 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.085129 23939 ts_tablet_manager.cc:1397] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250626 02:03:00.083310 23942 tablet_bootstrap.cc:654] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.085628 23939 raft_consensus.cc:357] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.085736 23939 raft_consensus.cc:383] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.085772 23939 raft_consensus.cc:738] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.085906 23879 tablet_service.cc:1468] Processing CreateTablet for tablet 70698cc805a448b28cef88e47e146427 (DEFAULT_TABLE table=test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250626 02:03:00.085932 23939 consensus_queue.cc:260] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.086134 23879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70698cc805a448b28cef88e47e146427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.086262 23939 ts_tablet_manager.cc:1428] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.086392 23940 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.086463 23940 ts_tablet_manager.cc:1397] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 02:03:00.086969 23942 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.087036 23942 ts_tablet_manager.cc:1397] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 02:03:00.087060 23940 raft_consensus.cc:357] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.087148 23940 raft_consensus.cc:383] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.087183 23940 raft_consensus.cc:738] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.087327 23940 consensus_queue.cc:260] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.087445 23942 raft_consensus.cc:357] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.087543 23942 raft_consensus.cc:383] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.087572 23942 raft_consensus.cc:738] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.087682 23940 ts_tablet_manager.cc:1428] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.087704 23942 consensus_queue.cc:260] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.088033 23942 ts_tablet_manager.cc:1428] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.088241 23942 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.088903 23883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13501c41bba94a7bbfb48a2715cd2d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:00.089344 23942 tablet_bootstrap.cc:654] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.089833 23940 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.090827 23942 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.090903 23942 ts_tablet_manager.cc:1397] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250626 02:03:00.091296 23942 raft_consensus.cc:357] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.091385 23942 raft_consensus.cc:383] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.091420 23942 raft_consensus.cc:738] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.086458 23939 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.091609 23942 consensus_queue.cc:260] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.091769 23940 tablet_bootstrap.cc:654] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.091965 23942 ts_tablet_manager.cc:1428] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.092182 23942 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.093037 23940 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.093150 23940 ts_tablet_manager.cc:1397] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:03:00.093246 23942 tablet_bootstrap.cc:654] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.093601 23940 raft_consensus.cc:357] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.093741 23940 raft_consensus.cc:383] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.093788 23940 raft_consensus.cc:738] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.093902 23940 consensus_queue.cc:260] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.094161 23940 ts_tablet_manager.cc:1428] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.094308 23942 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.094350 23940 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.094384 23942 ts_tablet_manager.cc:1397] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:00.094727 23942 raft_consensus.cc:357] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.094802 23942 raft_consensus.cc:383] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.094821 23942 raft_consensus.cc:738] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.094911 23942 consensus_queue.cc:260] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.095160 23942 ts_tablet_manager.cc:1428] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:00.095304 23940 tablet_bootstrap.cc:654] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.095304 23942 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.096258 23942 tablet_bootstrap.cc:654] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.096364 23939 tablet_bootstrap.cc:654] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.097329 23942 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.097425 23942 ts_tablet_manager.cc:1397] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:03:00.097440 23940 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.097513 23940 ts_tablet_manager.cc:1397] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:03:00.097836 23942 raft_consensus.cc:357] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.097935 23942 raft_consensus.cc:383] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.097975 23942 raft_consensus.cc:738] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.097329 23939 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.098166 23939 ts_tablet_manager.cc:1397] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20250626 02:03:00.097954 23940 raft_consensus.cc:357] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.098326 23940 raft_consensus.cc:383] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.098378 23940 raft_consensus.cc:738] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.098519 23940 consensus_queue.cc:260] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.098681 23939 raft_consensus.cc:357] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.098783 23940 ts_tablet_manager.cc:1428] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.098794 23939 raft_consensus.cc:383] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.098860 23939 raft_consensus.cc:738] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.098961 23940 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.098973 23939 consensus_queue.cc:260] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.099558 23939 ts_tablet_manager.cc:1428] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.099784 23939 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.099923 23940 tablet_bootstrap.cc:654] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.098163 23942 consensus_queue.cc:260] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.100436 23942 ts_tablet_manager.cc:1428] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250626 02:03:00.100639 23942 tablet_bootstrap.cc:492] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.100719 23939 tablet_bootstrap.cc:654] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.101629 23942 tablet_bootstrap.cc:654] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.101778 23939 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.101850 23939 ts_tablet_manager.cc:1397] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:03:00.102274 23939 raft_consensus.cc:357] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.102401 23939 raft_consensus.cc:383] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.102434 23939 raft_consensus.cc:738] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.102543 23939 consensus_queue.cc:260] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.102849 23939 ts_tablet_manager.cc:1428] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:00.103008 23939 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.103961 23939 tablet_bootstrap.cc:654] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.104677 23940 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.104722 23942 tablet_bootstrap.cc:492] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.104756 23940 ts_tablet_manager.cc:1397] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250626 02:03:00.104799 23942 ts_tablet_manager.cc:1397] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250626 02:03:00.105044 23939 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.105108 23939 ts_tablet_manager.cc:1397] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:03:00.105093 23942 raft_consensus.cc:357] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.105159 23942 raft_consensus.cc:383] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.105176 23940 raft_consensus.cc:357] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.105180 23942 raft_consensus.cc:738] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.105284 23940 raft_consensus.cc:383] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.105314 23940 raft_consensus.cc:738] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.105383 23942 consensus_queue.cc:260] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.105407 23940 consensus_queue.cc:260] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.105484 23939 raft_consensus.cc:357] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.105552 23939 raft_consensus.cc:383] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.105580 23939 raft_consensus.cc:738] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.105618 23942 ts_tablet_manager.cc:1428] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.105664 23940 ts_tablet_manager.cc:1428] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:00.105706 23939 consensus_queue.cc:260] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.105795 23942 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.105820 23940 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.106374 23939 ts_tablet_manager.cc:1428] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:00.106526 23939 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.106716 23942 tablet_bootstrap.cc:654] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.106722 23940 tablet_bootstrap.cc:654] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.107507 23939 tablet_bootstrap.cc:654] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.108598 23940 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.108598 23939 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.108705 23940 ts_tablet_manager.cc:1397] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:03:00.108705 23939 ts_tablet_manager.cc:1397] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250626 02:03:00.109160 23939 raft_consensus.cc:357] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.109287 23939 raft_consensus.cc:383] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.109328 23939 raft_consensus.cc:738] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.109455 23939 consensus_queue.cc:260] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.109596 23942 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.109699 23942 ts_tablet_manager.cc:1397] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250626 02:03:00.109714 23939 ts_tablet_manager.cc:1428] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:00.109870 23939 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.110073 23942 raft_consensus.cc:357] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.110190 23942 raft_consensus.cc:383] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.110235 23942 raft_consensus.cc:738] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.110370 23942 consensus_queue.cc:260] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.110683 23942 ts_tablet_manager.cc:1428] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.110785 23939 tablet_bootstrap.cc:654] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.110908 23942 tablet_bootstrap.cc:492] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.109160 23940 raft_consensus.cc:357] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.111380 23940 raft_consensus.cc:383] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.111418 23940 raft_consensus.cc:738] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.111552 23940 consensus_queue.cc:260] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.111805 23940 ts_tablet_manager.cc:1428] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250626 02:03:00.111825 23939 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.111905 23939 ts_tablet_manager.cc:1397] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250626 02:03:00.111917 23942 tablet_bootstrap.cc:654] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.111999 23940 tablet_bootstrap.cc:492] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.112375 23939 raft_consensus.cc:357] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.112475 23939 raft_consensus.cc:383] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.112510 23939 raft_consensus.cc:738] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.112674 23939 consensus_queue.cc:260] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.112979 23940 tablet_bootstrap.cc:654] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.113013 23939 ts_tablet_manager.cc:1428] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.113955 23939 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.115005 23939 tablet_bootstrap.cc:654] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.116115 23940 tablet_bootstrap.cc:492] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.116178 23939 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.116302 23940 ts_tablet_manager.cc:1397] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250626 02:03:00.116343 23939 ts_tablet_manager.cc:1397] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:03:00.116375 23942 tablet_bootstrap.cc:492] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.116464 23942 ts_tablet_manager.cc:1397] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250626 02:03:00.116818 23940 raft_consensus.cc:357] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.116853 23942 raft_consensus.cc:357] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.116829 23939 raft_consensus.cc:357] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.116941 23940 raft_consensus.cc:383] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.116971 23942 raft_consensus.cc:383] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.116988 23939 raft_consensus.cc:383] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.117015 23942 raft_consensus.cc:738] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.117044 23939 raft_consensus.cc:738] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.117170 23942 consensus_queue.cc:260] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.117172 23939 consensus_queue.cc:260] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.117501 23942 ts_tablet_manager.cc:1428] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:00.117544 23939 ts_tablet_manager.cc:1428] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.117669 23942 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.116993 23940 raft_consensus.cc:738] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.117875 23940 consensus_queue.cc:260] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.118297 23940 ts_tablet_manager.cc:1428] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:03:00.118492 23940 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.118650 23942 tablet_bootstrap.cc:654] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.117669 23939 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.119626 23940 tablet_bootstrap.cc:654] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.120244 23939 tablet_bootstrap.cc:654] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.120854 23940 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.120937 23940 ts_tablet_manager.cc:1397] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:03:00.121083 23942 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.121141 23942 ts_tablet_manager.cc:1397] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250626 02:03:00.121361 23940 raft_consensus.cc:357] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.121472 23940 raft_consensus.cc:383] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.121506 23940 raft_consensus.cc:738] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.121521 23942 raft_consensus.cc:357] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.121598 23942 raft_consensus.cc:383] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.121627 23942 raft_consensus.cc:738] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.121624 23940 consensus_queue.cc:260] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.121757 23942 consensus_queue.cc:260] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.121919 23940 ts_tablet_manager.cc:1428] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:00.121976 23942 ts_tablet_manager.cc:1428] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:00.122072 23940 tablet_bootstrap.cc:492] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.122112 23942 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.123015 23942 tablet_bootstrap.cc:654] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.123015 23940 tablet_bootstrap.cc:654] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.124193 23942 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.124312 23942 ts_tablet_manager.cc:1397] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:03:00.124532 23940 tablet_bootstrap.cc:492] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.124616 23940 ts_tablet_manager.cc:1397] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:03:00.124832 23942 raft_consensus.cc:357] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.124938 23942 raft_consensus.cc:383] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.124974 23942 raft_consensus.cc:738] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.125063 23940 raft_consensus.cc:357] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.125115 23942 consensus_queue.cc:260] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.125160 23940 raft_consensus.cc:383] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.125240 23940 raft_consensus.cc:738] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.125368 23940 consensus_queue.cc:260] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.125403 23942 ts_tablet_manager.cc:1428] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.125595 23942 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.125607 23940 ts_tablet_manager.cc:1428] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:00.125823 23940 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.126580 23942 tablet_bootstrap.cc:654] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.126767 23940 tablet_bootstrap.cc:654] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.127657 23942 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.127743 23942 ts_tablet_manager.cc:1397] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250626 02:03:00.127749 23940 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.127853 23940 ts_tablet_manager.cc:1397] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250626 02:03:00.128192 23942 raft_consensus.cc:357] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.128302 23942 raft_consensus.cc:383] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.128340 23939 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.128325 23940 raft_consensus.cc:357] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.128376 23942 raft_consensus.cc:738] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.128515 23939 ts_tablet_manager.cc:1397] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20250626 02:03:00.128515 23940 raft_consensus.cc:383] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.128643 23940 raft_consensus.cc:738] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.128619 23942 consensus_queue.cc:260] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.128803 23940 consensus_queue.cc:260] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.128917 23942 ts_tablet_manager.cc:1428] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.128984 23939 raft_consensus.cc:357] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.129099 23939 raft_consensus.cc:383] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.129122 23942 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.129585 23940 ts_tablet_manager.cc:1428] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:00.129799 23940 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.130174 23942 tablet_bootstrap.cc:654] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.129149 23939 raft_consensus.cc:738] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.130686 23939 consensus_queue.cc:260] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.131081 23939 ts_tablet_manager.cc:1428] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:03:00.131376 23939 tablet_bootstrap.cc:492] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.132097 23940 tablet_bootstrap.cc:654] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.132400 23939 tablet_bootstrap.cc:654] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.133271 23940 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.133356 23939 tablet_bootstrap.cc:492] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.133356 23940 ts_tablet_manager.cc:1397] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250626 02:03:00.133455 23939 ts_tablet_manager.cc:1397] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:00.133859 23939 raft_consensus.cc:357] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.133960 23939 raft_consensus.cc:383] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.133914 23940 raft_consensus.cc:357] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.134023 23939 raft_consensus.cc:738] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.134078 23940 raft_consensus.cc:383] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.134117 23940 raft_consensus.cc:738] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.134162 23942 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.134227 23940 consensus_queue.cc:260] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.134166 23939 consensus_queue.cc:260] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.134241 23942 ts_tablet_manager.cc:1397] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 02:03:00.134513 23940 ts_tablet_manager.cc:1428] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:00.134542 23939 ts_tablet_manager.cc:1428] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250626 02:03:00.134717 23940 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.134743 23942 raft_consensus.cc:357] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.134860 23942 raft_consensus.cc:383] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.134891 23942 raft_consensus.cc:738] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.135030 23942 consensus_queue.cc:260] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.135620 23939 tablet_bootstrap.cc:492] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.135802 23940 tablet_bootstrap.cc:654] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.135869 23942 ts_tablet_manager.cc:1428] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:00.136075 23942 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: Bootstrap starting.
I20250626 02:03:00.136601 23939 tablet_bootstrap.cc:654] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.137919 23942 tablet_bootstrap.cc:654] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.138372 23939 tablet_bootstrap.cc:492] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.138454 23939 ts_tablet_manager.cc:1397] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:03:00.138756 23939 raft_consensus.cc:357] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.138821 23939 raft_consensus.cc:383] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.138841 23939 raft_consensus.cc:738] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.138937 23939 consensus_queue.cc:260] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.139163 23939 ts_tablet_manager.cc:1428] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:00.139281 23939 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.140312 23939 tablet_bootstrap.cc:654] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.140954 23940 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.141010 23942 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: No bootstrap required, opened a new log
I20250626 02:03:00.141104 23942 ts_tablet_manager.cc:1397] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 02:03:00.141104 23940 ts_tablet_manager.cc:1397] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20250626 02:03:00.141597 23942 raft_consensus.cc:357] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.141585 23940 raft_consensus.cc:357] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.141741 23942 raft_consensus.cc:383] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.141750 23940 raft_consensus.cc:383] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.141788 23942 raft_consensus.cc:738] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.141801 23940 raft_consensus.cc:738] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.141829 23939 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.141891 23939 ts_tablet_manager.cc:1397] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:03:00.141902 23942 consensus_queue.cc:260] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.141902 23940 consensus_queue.cc:260] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.142202 23942 ts_tablet_manager.cc:1428] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.142200 23940 ts_tablet_manager.cc:1428] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:00.142282 23939 raft_consensus.cc:357] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.142369 23939 raft_consensus.cc:383] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.142400 23939 raft_consensus.cc:738] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.142549 23939 consensus_queue.cc:260] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.142766 23940 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4: Bootstrap starting.
I20250626 02:03:00.143764 23940 tablet_bootstrap.cc:654] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.144405 23939 ts_tablet_manager.cc:1428] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:03:00.144584 23939 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: Bootstrap starting.
I20250626 02:03:00.145606 23939 tablet_bootstrap.cc:654] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:00.145846 23940 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4: No bootstrap required, opened a new log
I20250626 02:03:00.145915 23940 ts_tablet_manager.cc:1397] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250626 02:03:00.146329 23940 raft_consensus.cc:357] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.146435 23940 raft_consensus.cc:383] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.146457 23940 raft_consensus.cc:738] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.146582 23940 consensus_queue.cc:260] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.146721 23939 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: No bootstrap required, opened a new log
I20250626 02:03:00.146821 23940 ts_tablet_manager.cc:1428] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:00.146839 23939 ts_tablet_manager.cc:1397] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250626 02:03:00.147214 23939 raft_consensus.cc:357] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.147301 23939 raft_consensus.cc:383] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:00.147343 23939 raft_consensus.cc:738] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Initialized, Role: FOLLOWER
I20250626 02:03:00.147460 23939 consensus_queue.cc:260] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.147783 23939 ts_tablet_manager.cc:1428] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:00.153091 23946 raft_consensus.cc:491] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.153179 23946 raft_consensus.cc:513] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.153650 23946 leader_election.cc:290] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.156905 23893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064" is_pre_election: true
I20250626 02:03:00.156984 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80" is_pre_election: true
I20250626 02:03:00.157030 23893 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 0.
I20250626 02:03:00.157120 23818 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 0.
I20250626 02:03:00.157305 23705 leader_election.cc:304] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.157492 23946 raft_consensus.cc:2802] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.157544 23946 raft_consensus.cc:491] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.157574 23946 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.158347 23946 raft_consensus.cc:513] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.158695 23946 leader_election.cc:290] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.159004 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:00.159025 23893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:00.159147 23818 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.159152 23893 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.160084 23893 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 1.
I20250626 02:03:00.160072 23818 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 1.
I20250626 02:03:00.160378 23705 leader_election.cc:304] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.160583 23946 raft_consensus.cc:2802] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.160797 23946 raft_consensus.cc:695] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:00.160924 23946 consensus_queue.cc:237] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.162463 23637 catalog_manager.cc:5582] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:00.190701 23945 raft_consensus.cc:491] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.190905 23945 raft_consensus.cc:513] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.191406 23945 leader_election.cc:290] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.194921 23893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064" is_pre_election: true
I20250626 02:03:00.195082 23893 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 0.
I20250626 02:03:00.195199 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4" is_pre_election: true
I20250626 02:03:00.195320 23742 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 0.
I20250626 02:03:00.195434 23781 leader_election.cc:304] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.195607 23945 raft_consensus.cc:2802] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.195662 23945 raft_consensus.cc:491] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.195708 23945 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.196563 23945 raft_consensus.cc:513] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.196882 23945 leader_election.cc:290] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.197172 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:00.197188 23893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:00.197278 23742 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.197290 23893 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.198029 23893 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 1.
I20250626 02:03:00.198071 23742 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 1.
I20250626 02:03:00.198295 23781 leader_election.cc:304] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.198443 23945 raft_consensus.cc:2802] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.198678 23945 raft_consensus.cc:695] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:00.198827 23945 consensus_queue.cc:237] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.200335 23637 catalog_manager.cc:5582] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:00.227474 23945 raft_consensus.cc:491] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.227602 23945 raft_consensus.cc:513] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.227983 23945 leader_election.cc:290] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.228395 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "894851ab3c1e4d79971597e62b99cfbf" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4" is_pre_election: true
I20250626 02:03:00.228425 23893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "894851ab3c1e4d79971597e62b99cfbf" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064" is_pre_election: true
I20250626 02:03:00.228540 23742 raft_consensus.cc:2466] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 0.
I20250626 02:03:00.228565 23893 raft_consensus.cc:2466] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 0.
I20250626 02:03:00.228818 23784 leader_election.cc:304] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 6c2126b790c0477595d5b94c658b8a80; no voters:
I20250626 02:03:00.229002 23945 raft_consensus.cc:2802] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.229061 23945 raft_consensus.cc:491] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.229107 23945 raft_consensus.cc:3058] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.229962 23945 raft_consensus.cc:513] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.230293 23945 leader_election.cc:290] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.230676 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "894851ab3c1e4d79971597e62b99cfbf" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:00.230677 23893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "894851ab3c1e4d79971597e62b99cfbf" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:00.230813 23742 raft_consensus.cc:3058] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.230827 23893 raft_consensus.cc:3058] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.230849 23946 raft_consensus.cc:491] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.230911 23946 raft_consensus.cc:513] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.231256 23946 leader_election.cc:290] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.231555 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "363d650f886244d38b78e6af7bb5f6ce" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80" is_pre_election: true
I20250626 02:03:00.231662 23818 raft_consensus.cc:2466] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 0.
I20250626 02:03:00.231789 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "363d650f886244d38b78e6af7bb5f6ce" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064" is_pre_election: true
I20250626 02:03:00.231792 23893 raft_consensus.cc:2466] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 1.
I20250626 02:03:00.231889 23892 raft_consensus.cc:2466] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 0.
I20250626 02:03:00.231942 23707 leader_election.cc:304] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 6c2126b790c0477595d5b94c658b8a80; no voters:
I20250626 02:03:00.232092 23781 leader_election.cc:304] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.232095 23946 raft_consensus.cc:2802] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.232215 23946 raft_consensus.cc:491] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.232252 23946 raft_consensus.cc:3058] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.232280 23945 raft_consensus.cc:2802] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.232283 23742 raft_consensus.cc:2466] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 1.
I20250626 02:03:00.232419 23945 raft_consensus.cc:695] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:00.232554 23945 consensus_queue.cc:237] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.232973 23946 raft_consensus.cc:513] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.233264 23946 leader_election.cc:290] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.233489 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "363d650f886244d38b78e6af7bb5f6ce" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:00.233579 23818 raft_consensus.cc:3058] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.233978 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "363d650f886244d38b78e6af7bb5f6ce" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:00.234095 23892 raft_consensus.cc:3058] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.234359 23947 raft_consensus.cc:491] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.234443 23947 raft_consensus.cc:513] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.234773 23947 leader_election.cc:290] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:00.234889 23818 raft_consensus.cc:2466] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 1.
I20250626 02:03:00.235039 23892 raft_consensus.cc:2466] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 1.
I20250626 02:03:00.235284 23705 leader_election.cc:304] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.236399 23637 catalog_manager.cc:5582] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:00.237771 23946 raft_consensus.cc:2802] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.237848 23946 raft_consensus.cc:695] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:00.237944 23946 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.238373 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80" is_pre_election: true
I20250626 02:03:00.238476 23818 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 0.
I20250626 02:03:00.238759 23858 leader_election.cc:304] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.238920 23947 raft_consensus.cc:2802] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.238965 23947 raft_consensus.cc:491] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.239001 23947 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.239472 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4" is_pre_election: true
I20250626 02:03:00.239571 23742 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 0.
I20250626 02:03:00.239768 23947 raft_consensus.cc:513] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.239748 23638 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:00.240041 23947 leader_election.cc:290] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:00.240293 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:00.240386 23742 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.240511 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:00.240615 23818 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.241065 23742 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 1.
I20250626 02:03:00.241281 23818 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 1.
I20250626 02:03:00.241349 23859 leader_election.cc:304] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.241526 23947 raft_consensus.cc:2802] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.241729 23947 raft_consensus.cc:695] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Becoming Leader. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:00.241875 23947 consensus_queue.cc:237] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.243305 23638 catalog_manager.cc:5582] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 reported cstate change: term changed from 0 to 1, leader changed from <none> to d150470bce35446b9b3f41ec78cb5064 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:00.278508 23947 raft_consensus.cc:491] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.278666 23947 raft_consensus.cc:513] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.279054 23947 leader_election.cc:290] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:00.279476 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ac2721a9bb44efa815122997c53fd7" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4" is_pre_election: true
I20250626 02:03:00.279479 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ac2721a9bb44efa815122997c53fd7" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80" is_pre_election: true
I20250626 02:03:00.279626 23742 raft_consensus.cc:2466] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 0.
I20250626 02:03:00.279665 23818 raft_consensus.cc:2466] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 0.
I20250626 02:03:00.279898 23859 leader_election.cc:304] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.280098 23947 raft_consensus.cc:2802] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.280177 23947 raft_consensus.cc:491] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.280220 23947 raft_consensus.cc:3058] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.281261 23993 raft_consensus.cc:491] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.281368 23993 raft_consensus.cc:513] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.281495 23947 raft_consensus.cc:513] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.281759 23993 leader_election.cc:290] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:00.281761 23947 leader_election.cc:290] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:00.282142 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ac2721a9bb44efa815122997c53fd7" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:00.282246 23742 raft_consensus.cc:3058] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.282251 23741 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4" is_pre_election: true
I20250626 02:03:00.282285 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ac2721a9bb44efa815122997c53fd7" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:00.282368 23741 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 0.
I20250626 02:03:00.282285 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80" is_pre_election: true
I20250626 02:03:00.282629 23817 raft_consensus.cc:3058] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.282745 23818 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 0.
I20250626 02:03:00.282841 23859 leader_election.cc:304] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.283294 23742 raft_consensus.cc:2466] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 1.
I20250626 02:03:00.283602 23947 raft_consensus.cc:2802] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.283584 23859 leader_election.cc:304] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.283596 23817 raft_consensus.cc:2466] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 1.
I20250626 02:03:00.283686 23947 raft_consensus.cc:491] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.283980 23993 raft_consensus.cc:2802] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.284009 23947 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.284229 23993 raft_consensus.cc:695] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Becoming Leader. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:00.284364 23993 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.286365 23638 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: term changed from 0 to 1, leader changed from <none> to d150470bce35446b9b3f41ec78cb5064 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:00.287259 23947 raft_consensus.cc:513] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.287603 23947 leader_election.cc:290] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:00.288071 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:00.288198 23742 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.288957 23742 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 1.
I20250626 02:03:00.289160 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:00.289252 23817 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.289249 23859 leader_election.cc:304] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.289633 23947 raft_consensus.cc:2802] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.289716 23947 raft_consensus.cc:695] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Becoming Leader. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:00.289835 23947 consensus_queue.cc:237] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.290148 23817 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 1.
I20250626 02:03:00.291759 23638 catalog_manager.cc:5582] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: term changed from 0 to 1, leader changed from <none> to d150470bce35446b9b3f41ec78cb5064 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:00.315881 23987 raft_consensus.cc:491] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.316040 23987 raft_consensus.cc:513] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.316601 23987 leader_election.cc:290] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.316980 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064" is_pre_election: true
I20250626 02:03:00.316982 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4" is_pre_election: true
I20250626 02:03:00.317121 23892 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 0.
I20250626 02:03:00.317189 23742 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 0.
I20250626 02:03:00.317404 23781 leader_election.cc:304] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.317553 23987 raft_consensus.cc:2802] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.317610 23987 raft_consensus.cc:491] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.317644 23987 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.318485 23987 raft_consensus.cc:513] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.318830 23987 leader_election.cc:290] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.319136 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:00.319164 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:00.319269 23742 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.319281 23892 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.320093 23892 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 1.
I20250626 02:03:00.320418 23781 leader_election.cc:304] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.320461 23742 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 1.
I20250626 02:03:00.320583 23987 raft_consensus.cc:2802] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.320653 23987 raft_consensus.cc:695] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:00.320834 23987 consensus_queue.cc:237] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.322496 23638 catalog_manager.cc:5582] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:00.331576 23947 raft_consensus.cc:491] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.331712 23947 raft_consensus.cc:513] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.332098 23947 leader_election.cc:290] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:00.332468 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e37650dfb570400fb02742cb1220c7e1" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80" is_pre_election: true
I20250626 02:03:00.332465 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e37650dfb570400fb02742cb1220c7e1" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4" is_pre_election: true
I20250626 02:03:00.332587 23817 raft_consensus.cc:2466] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 0.
I20250626 02:03:00.332608 23742 raft_consensus.cc:2466] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 0.
I20250626 02:03:00.332859 23858 leader_election.cc:304] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.333060 23947 raft_consensus.cc:2802] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.333115 23947 raft_consensus.cc:491] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.333153 23947 raft_consensus.cc:3058] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.333913 23947 raft_consensus.cc:513] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.334188 23947 leader_election.cc:290] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:00.334493 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e37650dfb570400fb02742cb1220c7e1" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:00.334511 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e37650dfb570400fb02742cb1220c7e1" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:00.334625 23817 raft_consensus.cc:3058] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.334625 23742 raft_consensus.cc:3058] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.335506 23817 raft_consensus.cc:2466] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 1.
I20250626 02:03:00.335506 23742 raft_consensus.cc:2466] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 1.
I20250626 02:03:00.335845 23859 leader_election.cc:304] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.335980 23947 raft_consensus.cc:2802] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.336086 23947 raft_consensus.cc:695] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Becoming Leader. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:00.336231 23947 consensus_queue.cc:237] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.337636 23638 catalog_manager.cc:5582] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: term changed from 0 to 1, leader changed from <none> to d150470bce35446b9b3f41ec78cb5064 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:00.345606 23983 raft_consensus.cc:491] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.345705 23983 raft_consensus.cc:513] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.346092 23983 leader_election.cc:290] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.346412 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80" is_pre_election: true
I20250626 02:03:00.346413 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064" is_pre_election: true
I20250626 02:03:00.346539 23817 raft_consensus.cc:2466] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 0.
I20250626 02:03:00.346544 23892 raft_consensus.cc:2466] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 0.
I20250626 02:03:00.346796 23705 leader_election.cc:304] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.346943 23983 raft_consensus.cc:2802] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.347004 23983 raft_consensus.cc:491] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.347059 23983 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.348165 23983 raft_consensus.cc:513] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.348518 23983 leader_election.cc:290] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.348879 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:00.348904 23945 raft_consensus.cc:491] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.348979 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:00.349030 23945 raft_consensus.cc:513] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.349071 23892 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.349030 23817 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.349368 23945 leader_election.cc:290] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.349694 23893 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064" is_pre_election: true
I20250626 02:03:00.349776 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4" is_pre_election: true
I20250626 02:03:00.349805 23893 raft_consensus.cc:2466] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 0.
I20250626 02:03:00.349854 23742 raft_consensus.cc:2466] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 0.
I20250626 02:03:00.350013 23892 raft_consensus.cc:2466] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 1.
I20250626 02:03:00.350013 23817 raft_consensus.cc:2466] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 1.
I20250626 02:03:00.350086 23781 leader_election.cc:304] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.350234 23945 raft_consensus.cc:2802] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.350286 23945 raft_consensus.cc:491] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.350327 23945 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.350349 23705 leader_election.cc:304] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.350471 23983 raft_consensus.cc:2802] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.350538 23983 raft_consensus.cc:695] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:00.350697 23983 consensus_queue.cc:237] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.351187 23945 raft_consensus.cc:513] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.351490 23945 leader_election.cc:290] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.351863 23742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:00.351966 23742 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.352013 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:00.352087 23892 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.352293 23638 catalog_manager.cc:5582] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:00.352803 23742 raft_consensus.cc:2466] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 1.
I20250626 02:03:00.352885 23892 raft_consensus.cc:2466] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 1.
I20250626 02:03:00.353098 23784 leader_election.cc:304] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 6c2126b790c0477595d5b94c658b8a80; no voters:
I20250626 02:03:00.353232 23945 raft_consensus.cc:2802] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.353330 23945 raft_consensus.cc:695] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:00.353470 23945 consensus_queue.cc:237] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.354831 23638 catalog_manager.cc:5582] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:00.411087 23983 raft_consensus.cc:491] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:00.411245 23983 raft_consensus.cc:513] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.411708 23983 leader_election.cc:290] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.412231 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a4a1bceeb5a4e3faba6c6ee9f5ec534" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064" is_pre_election: true
I20250626 02:03:00.412236 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a4a1bceeb5a4e3faba6c6ee9f5ec534" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80" is_pre_election: true
I20250626 02:03:00.412379 23892 raft_consensus.cc:2466] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 0.
I20250626 02:03:00.412398 23817 raft_consensus.cc:2466] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 0.
I20250626 02:03:00.412678 23705 leader_election.cc:304] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.412830 23983 raft_consensus.cc:2802] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:00.412890 23983 raft_consensus.cc:491] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:00.412945 23983 raft_consensus.cc:3058] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.413772 23983 raft_consensus.cc:513] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.414105 23983 leader_election.cc:290] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:00.414417 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a4a1bceeb5a4e3faba6c6ee9f5ec534" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:00.414426 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a4a1bceeb5a4e3faba6c6ee9f5ec534" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:00.414512 23892 raft_consensus.cc:3058] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.414542 23817 raft_consensus.cc:3058] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:00.415287 23892 raft_consensus.cc:2466] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 1.
I20250626 02:03:00.415295 23817 raft_consensus.cc:2466] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 1.
I20250626 02:03:00.415552 23705 leader_election.cc:304] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:00.415704 23983 raft_consensus.cc:2802] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:00.415791 23983 raft_consensus.cc:695] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:00.415920 23983 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:00.417490 23638 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:00.471771 23817 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.472604 23892 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.472935 23893 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.473052 23742 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.473227 23740 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.473294 23947 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.473352 23741 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.473227 23891 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.474419 23987 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.474696 23987 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.474817 23891 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.474938 23987 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.475155 23987 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.475373 23987 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.475666 23987 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.475901 23987 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.476707 23818 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.476986 23741 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.477766 23818 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.478113 23947 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.479025 23738 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:03:00.479900 23738 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.479909 23737 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.480168 23947 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.480569 23741 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.484854 23947 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.485350 24032 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.485618 23995 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.485796 24032 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.493901 24020 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.496790 23818 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.500490 23889 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.500900 23814 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.501927 23888 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.502362 23813 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.504508 23887 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.504725 23886 raft_consensus.cc:1273] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.504976 23812 raft_consensus.cc:1273] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.510270 24014 mvcc.cc:204] Tried to move back new op lower bound from 7171700246444666880 to 7171700245858500608. Current Snapshot: MvccSnapshot[applied={T|T < 7171700246444666880}]
I20250626 02:03:00.511700 23811 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:00.517526 23946 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.517859 23946 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.518112 23946 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.518370 23946 consensus_queue.cc:1035] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.523944 23995 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.524394 23983 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.524627 23983 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.524904 23983 consensus_queue.cc:1035] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.525143 23983 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:00.535090 24024 mvcc.cc:204] Tried to move back new op lower bound from 7171700246549700608 to 7171700246185115648. Current Snapshot: MvccSnapshot[applied={T|T < 7171700246532296704}]
I20250626 02:03:00.549573 24016 mvcc.cc:204] Tried to move back new op lower bound from 7171700246450073600 to 7171700245471993856. Current Snapshot: MvccSnapshot[applied={T|T < 7171700246432587776}]
I20250626 02:03:00.860118 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:00.861826 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:00.861928 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:00.981863 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:03:00.983554 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:00.983592 24089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:00.983654 24087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:00.983676 24086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:00.983983 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:00.984030 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:00.984045 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903380984045 us; error 0 us; skew 500 ppm
I20250626 02:03:00.985041 20338 webserver.cc:469] Webserver started at http://127.19.220.132:42107/ using document root <none> and password file <none>
I20250626 02:03:00.985184 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:00.985252 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:00.985327 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:00.985697 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-3-root/instance:
uuid: "145a19f8089541e3845de3dc9a3ac9dd"
format_stamp: "Formatted at 2025-06-26 02:03:00 on dist-test-slave-h18f"
I20250626 02:03:00.987093 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:03:00.987895 24094 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:00.988107 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:00.988210 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-3-root
uuid: "145a19f8089541e3845de3dc9a3ac9dd"
format_stamp: "Formatted at 2025-06-26 02:03:00 on dist-test-slave-h18f"
I20250626 02:03:00.988276 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:01.013358 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:01.013761 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:01.014039 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:01.014469 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:01.014508 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:01.014557 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:01.014595 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:01.019011 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.132:33409
I20250626 02:03:01.019061 24157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.132:33409 every 8 connection(s)
I20250626 02:03:01.019865 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:01.021664 24165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:01.021812 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:01.022015 24162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:01.022222 24163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:01.022428 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:01.022490 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:01.022512 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903381022512 us; error 0 us; skew 500 ppm
I20250626 02:03:01.023916 20338 webserver.cc:469] Webserver started at http://127.19.220.133:41347/ using document root <none> and password file <none>
I20250626 02:03:01.024030 24158 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42171
I20250626 02:03:01.024065 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:01.024112 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:01.024163 24158 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:01.024268 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:01.024449 24158 heartbeater.cc:507] Master 127.19.220.190:42171 requested a full tablet report, sending...
I20250626 02:03:01.024716 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-4-root/instance:
uuid: "c5ee348337b146838a15e2d7161f5fb8"
format_stamp: "Formatted at 2025-06-26 02:03:01 on dist-test-slave-h18f"
I20250626 02:03:01.025111 23633 ts_manager.cc:194] Registered new tserver with Master: 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409)
I20250626 02:03:01.025830 23633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41636
I20250626 02:03:01.026422 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:03:01.027318 24170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:01.027542 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:01.027601 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-4-root
uuid: "c5ee348337b146838a15e2d7161f5fb8"
format_stamp: "Formatted at 2025-06-26 02:03:01 on dist-test-slave-h18f"
I20250626 02:03:01.027653 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:01.059885 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:01.060367 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:01.060649 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:01.061100 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:01.061138 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:01.061169 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:01.061203 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:01.065937 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.133:43947
I20250626 02:03:01.066061 24233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.133:43947 every 8 connection(s)
I20250626 02:03:01.066818 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:01.068598 24237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:01.068634 24238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:01.068775 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:01.068923 24240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:01.069124 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:01.069171 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:01.069205 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903381069204 us; error 0 us; skew 500 ppm
I20250626 02:03:01.070302 20338 webserver.cc:469] Webserver started at http://127.19.220.134:43109/ using document root <none> and password file <none>
I20250626 02:03:01.070456 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:01.070506 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:01.070627 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:01.071051 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-5-root/instance:
uuid: "46dfe87116554377a3d4bf78c9f9699b"
format_stamp: "Formatted at 2025-06-26 02:03:01 on dist-test-slave-h18f"
I20250626 02:03:01.071345 24234 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42171
I20250626 02:03:01.071448 24234 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:01.071650 24234 heartbeater.cc:507] Master 127.19.220.190:42171 requested a full tablet report, sending...
I20250626 02:03:01.072288 23633 ts_manager.cc:194] Registered new tserver with Master: c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947)
I20250626 02:03:01.072827 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250626 02:03:01.072901 23633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41650
I20250626 02:03:01.073810 24245 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:01.074074 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:01.074134 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-5-root
uuid: "46dfe87116554377a3d4bf78c9f9699b"
format_stamp: "Formatted at 2025-06-26 02:03:01 on dist-test-slave-h18f"
I20250626 02:03:01.074189 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:01.088536 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:01.088950 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:01.089283 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:01.089761 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:01.089800 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:01.089831 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:01.089852 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:01.094328 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.134:42101
I20250626 02:03:01.094421 24308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.134:42101 every 8 connection(s)
I20250626 02:03:01.098738 24309 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42171
I20250626 02:03:01.098842 24309 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:01.099057 24309 heartbeater.cc:507] Master 127.19.220.190:42171 requested a full tablet report, sending...
I20250626 02:03:01.099635 23633 ts_manager.cc:194] Registered new tserver with Master: 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101)
I20250626 02:03:01.100175 23633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41662
I20250626 02:03:01.869282 23818 consensus_queue.cc:237] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:01.870729 23891 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 02:03:01.870919 23739 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 02:03:01.871165 24316 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:01.871327 24315 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:01.873036 24316 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:01.873433 23891 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:01.873433 23739 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
W20250626 02:03:01.875633 23784 consensus_peers.cc:487] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 894851ab3c1e4d79971597e62b99cfbf. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:01.875931 23633 catalog_manager.cc:5582] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index -1 to 28, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:01.880352 23891 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:01.881556 23818 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 02:03:01.881603 23739 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 02:03:01.881897 24324 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:01.882055 23993 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:01.883682 23818 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:01.883709 23739 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:01.883333 24325 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:01.886291 23635 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index -1 to 28, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
W20250626 02:03:01.886420 23856 consensus_peers.cc:487] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 -> Peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101): Couldn't send request to peer 46dfe87116554377a3d4bf78c9f9699b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34ac2721a9bb44efa815122997c53fd7. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:01.890338 23739 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:01.891557 23891 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:01.891716 23818 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:01.891878 24312 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:01.892516 24312 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:01.893580 24060 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:01.894121 23891 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:01.894156 23818 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:01.896199 23633 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index -1 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:01.896606 23817 consensus_queue.cc:237] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
W20250626 02:03:01.897301 23706 consensus_peers.cc:487] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 -> Peer c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947): Couldn't send request to peer c5ee348337b146838a15e2d7161f5fb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 363d650f886244d38b78e6af7bb5f6ce. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:01.897857 23891 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:01.898384 24316 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:01.898707 23739 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:01.898980 24316 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:01.899693 24017 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:01.901021 23739 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:01.901502 23891 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:01.901837 23892 consensus_queue.cc:237] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
W20250626 02:03:01.902884 23782 consensus_peers.cc:487] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 -> Peer c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947): Couldn't send request to peer c5ee348337b146838a15e2d7161f5fb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65bf2a25991b4d4a94efbc900569a162. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:01.903636 23739 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250626 02:03:01.903998 23993 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250626 02:03:01.904084 23639 catalog_manager.cc:5582] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index -1 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:01.905599 23817 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250626 02:03:01.905602 23993 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:01.906016 23739 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
W20250626 02:03:01.906935 23859 consensus_peers.cc:487] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e37650dfb570400fb02742cb1220c7e1. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:01.908056 23739 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:01.908427 23635 catalog_manager.cc:5582] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index -1 to 26, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:01.909391 23891 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 02:03:01.909631 23817 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 02:03:01.909725 24325 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250626 02:03:01.910169 24329 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:01.911376 23818 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:01.911604 24329 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:01.912810 24329 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20250626 02:03:01.912979 23705 consensus_peers.cc:487] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 -> Peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101): Couldn't send request to peer 46dfe87116554377a3d4bf78c9f9699b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4a1bceeb5a4e3faba6c6ee9f5ec534. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:01.913797 23891 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:01.913971 23633 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index -1 to 28, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:01.914985 23818 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:02.026746 24158 heartbeater.cc:499] Master 127.19.220.190:42171 was elected leader, sending a full tablet report...
I20250626 02:03:02.073635 24234 heartbeater.cc:499] Master 127.19.220.190:42171 was elected leader, sending a full tablet report...
I20250626 02:03:02.100958 24309 heartbeater.cc:499] Master 127.19.220.190:42171 was elected leader, sending a full tablet report...
I20250626 02:03:02.287662 24333 ts_tablet_manager.cc:927] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: Initiating tablet copy from peer 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:02.288033 24333 tablet_copy_client.cc:323] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:39183
I20250626 02:03:02.291855 24336 ts_tablet_manager.cc:927] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: Initiating tablet copy from peer 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725)
I20250626 02:03:02.291811 23828 tablet_copy_service.cc:140] P 6c2126b790c0477595d5b94c658b8a80: Received BeginTabletCopySession request for tablet 894851ab3c1e4d79971597e62b99cfbf from peer 145a19f8089541e3845de3dc9a3ac9dd ({username='slave'} at 127.0.0.1:52044)
I20250626 02:03:02.291980 23828 tablet_copy_service.cc:161] P 6c2126b790c0477595d5b94c658b8a80: Beginning new tablet copy session on tablet 894851ab3c1e4d79971597e62b99cfbf from peer 145a19f8089541e3845de3dc9a3ac9dd at {username='slave'} at 127.0.0.1:52044: session id = 145a19f8089541e3845de3dc9a3ac9dd-894851ab3c1e4d79971597e62b99cfbf
I20250626 02:03:02.292194 24336 tablet_copy_client.cc:323] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:34725
I20250626 02:03:02.293218 23828 tablet_copy_source_session.cc:215] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:02.293664 24339 ts_tablet_manager.cc:927] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: Initiating tablet copy from peer d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:02.294008 24333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894851ab3c1e4d79971597e62b99cfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:02.294018 24339 tablet_copy_client.cc:323] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:40133
I20250626 02:03:02.295331 24341 ts_tablet_manager.cc:927] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: Initiating tablet copy from peer 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725)
I20250626 02:03:02.295617 24341 tablet_copy_client.cc:323] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:34725
I20250626 02:03:02.296469 24333 tablet_copy_client.cc:806] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:02.296624 24333 tablet_copy_client.cc:670] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:02.297654 23752 tablet_copy_service.cc:140] P 3a8897880da64721ad84d66ff1fcf3a4: Received BeginTabletCopySession request for tablet 363d650f886244d38b78e6af7bb5f6ce from peer c5ee348337b146838a15e2d7161f5fb8 ({username='slave'} at 127.0.0.1:40626)
I20250626 02:03:02.297729 24334 tablet_copy_client.cc:1025] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:02.297767 23752 tablet_copy_service.cc:161] P 3a8897880da64721ad84d66ff1fcf3a4: Beginning new tablet copy session on tablet 363d650f886244d38b78e6af7bb5f6ce from peer c5ee348337b146838a15e2d7161f5fb8 at {username='slave'} at 127.0.0.1:40626: session id = c5ee348337b146838a15e2d7161f5fb8-363d650f886244d38b78e6af7bb5f6ce
I20250626 02:03:02.298454 23752 tablet_copy_source_session.cc:215] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:02.299299 24336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363d650f886244d38b78e6af7bb5f6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:02.301364 23752 tablet_copy_service.cc:140] P 3a8897880da64721ad84d66ff1fcf3a4: Received BeginTabletCopySession request for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 from peer 46dfe87116554377a3d4bf78c9f9699b ({username='slave'} at 127.0.0.1:40636)
I20250626 02:03:02.301474 23752 tablet_copy_service.cc:161] P 3a8897880da64721ad84d66ff1fcf3a4: Beginning new tablet copy session on tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 from peer 46dfe87116554377a3d4bf78c9f9699b at {username='slave'} at 127.0.0.1:40636: session id = 46dfe87116554377a3d4bf78c9f9699b-3a4a1bceeb5a4e3faba6c6ee9f5ec534
I20250626 02:03:02.301506 24336 tablet_copy_client.cc:806] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:02.301638 24336 tablet_copy_client.cc:670] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:02.302328 23752 tablet_copy_source_session.cc:215] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:02.302768 24337 tablet_copy_client.cc:1025] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:02.303027 23903 tablet_copy_service.cc:140] P d150470bce35446b9b3f41ec78cb5064: Received BeginTabletCopySession request for tablet 34ac2721a9bb44efa815122997c53fd7 from peer 46dfe87116554377a3d4bf78c9f9699b ({username='slave'} at 127.0.0.1:37572)
I20250626 02:03:02.303107 23903 tablet_copy_service.cc:161] P d150470bce35446b9b3f41ec78cb5064: Beginning new tablet copy session on tablet 34ac2721a9bb44efa815122997c53fd7 from peer 46dfe87116554377a3d4bf78c9f9699b at {username='slave'} at 127.0.0.1:37572: session id = 46dfe87116554377a3d4bf78c9f9699b-34ac2721a9bb44efa815122997c53fd7
I20250626 02:03:02.303444 24341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:02.303819 23903 tablet_copy_source_session.cc:215] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:02.304589 24339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac2721a9bb44efa815122997c53fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:02.306298 24341 tablet_copy_client.cc:806] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:02.306460 24341 tablet_copy_client.cc:670] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:02.306509 24339 tablet_copy_client.cc:806] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:02.306634 24339 tablet_copy_client.cc:670] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:02.307348 24342 tablet_copy_client.cc:1025] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:02.307578 24340 tablet_copy_client.cc:1025] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:02.316546 24346 ts_tablet_manager.cc:927] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: Initiating tablet copy from peer d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:02.316924 24346 tablet_copy_client.cc:323] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:40133
I20250626 02:03:02.320155 23903 tablet_copy_service.cc:140] P d150470bce35446b9b3f41ec78cb5064: Received BeginTabletCopySession request for tablet e37650dfb570400fb02742cb1220c7e1 from peer 145a19f8089541e3845de3dc9a3ac9dd ({username='slave'} at 127.0.0.1:37576)
I20250626 02:03:02.320247 23903 tablet_copy_service.cc:161] P d150470bce35446b9b3f41ec78cb5064: Beginning new tablet copy session on tablet e37650dfb570400fb02742cb1220c7e1 from peer 145a19f8089541e3845de3dc9a3ac9dd at {username='slave'} at 127.0.0.1:37576: session id = 145a19f8089541e3845de3dc9a3ac9dd-e37650dfb570400fb02742cb1220c7e1
I20250626 02:03:02.320932 23903 tablet_copy_source_session.cc:215] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:02.321627 24346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37650dfb570400fb02742cb1220c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:02.323393 24346 tablet_copy_client.cc:806] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:02.323537 24346 tablet_copy_client.cc:670] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:02.324404 24347 tablet_copy_client.cc:1025] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:02.460762 24348 ts_tablet_manager.cc:927] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: Initiating tablet copy from peer 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:02.461125 24348 tablet_copy_client.cc:323] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:39183
I20250626 02:03:02.464505 23828 tablet_copy_service.cc:140] P 6c2126b790c0477595d5b94c658b8a80: Received BeginTabletCopySession request for tablet 65bf2a25991b4d4a94efbc900569a162 from peer c5ee348337b146838a15e2d7161f5fb8 ({username='slave'} at 127.0.0.1:52060)
I20250626 02:03:02.464599 23828 tablet_copy_service.cc:161] P 6c2126b790c0477595d5b94c658b8a80: Beginning new tablet copy session on tablet 65bf2a25991b4d4a94efbc900569a162 from peer c5ee348337b146838a15e2d7161f5fb8 at {username='slave'} at 127.0.0.1:52060: session id = c5ee348337b146838a15e2d7161f5fb8-65bf2a25991b4d4a94efbc900569a162
I20250626 02:03:02.465528 23828 tablet_copy_source_session.cc:215] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:02.466334 24348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65bf2a25991b4d4a94efbc900569a162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:02.468602 24348 tablet_copy_client.cc:806] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:02.468806 24348 tablet_copy_client.cc:670] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:02.469926 24349 tablet_copy_client.cc:1025] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:02.862159 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:02.863407 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:02.863528 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:04.298223 24333 tablet_copy_client.cc:538] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:04.299827 24333 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap starting.
I20250626 02:03:04.303210 24336 tablet_copy_client.cc:538] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:04.304662 24336 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: Bootstrap starting.
I20250626 02:03:04.307775 24341 tablet_copy_client.cc:538] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:04.307950 24339 tablet_copy_client.cc:538] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:04.309252 24341 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap starting.
I20250626 02:03:04.309322 24339 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap starting.
I20250626 02:03:04.324786 24346 tablet_copy_client.cc:538] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:04.326196 24346 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap starting.
I20250626 02:03:04.330904 24336 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:04.331162 24333 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:04.331265 24336 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: Bootstrap complete.
I20250626 02:03:04.331578 24336 ts_tablet_manager.cc:1397] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.011s
I20250626 02:03:04.331597 24333 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap complete.
I20250626 02:03:04.331861 24333 ts_tablet_manager.cc:1397] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.000s
I20250626 02:03:04.332458 24333 raft_consensus.cc:357] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:04.332666 24333 raft_consensus.cc:738] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 145a19f8089541e3845de3dc9a3ac9dd, State: Initialized, Role: LEARNER
I20250626 02:03:04.332811 24333 consensus_queue.cc:260] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:04.333240 24333 ts_tablet_manager.cc:1428] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:04.333688 23828 tablet_copy_service.cc:342] P 6c2126b790c0477595d5b94c658b8a80: Request end of tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-894851ab3c1e4d79971597e62b99cfbf received from {username='slave'} at 127.0.0.1:52044
I20250626 02:03:04.333770 23828 tablet_copy_service.cc:434] P 6c2126b790c0477595d5b94c658b8a80: ending tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-894851ab3c1e4d79971597e62b99cfbf on tablet 894851ab3c1e4d79971597e62b99cfbf with peer 145a19f8089541e3845de3dc9a3ac9dd
I20250626 02:03:04.334580 24336 raft_consensus.cc:357] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:04.334743 24336 raft_consensus.cc:738] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5ee348337b146838a15e2d7161f5fb8, State: Initialized, Role: LEARNER
I20250626 02:03:04.334879 24336 consensus_queue.cc:260] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:04.335240 24336 ts_tablet_manager.cc:1428] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 02:03:04.335965 23752 tablet_copy_service.cc:342] P 3a8897880da64721ad84d66ff1fcf3a4: Request end of tablet copy session c5ee348337b146838a15e2d7161f5fb8-363d650f886244d38b78e6af7bb5f6ce received from {username='slave'} at 127.0.0.1:40626
I20250626 02:03:04.336035 23752 tablet_copy_service.cc:434] P 3a8897880da64721ad84d66ff1fcf3a4: ending tablet copy session c5ee348337b146838a15e2d7161f5fb8-363d650f886244d38b78e6af7bb5f6ce on tablet 363d650f886244d38b78e6af7bb5f6ce with peer c5ee348337b146838a15e2d7161f5fb8
I20250626 02:03:04.336558 24208 raft_consensus.cc:1215] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20250626 02:03:04.346071 24339 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:04.346418 24339 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap complete.
I20250626 02:03:04.346647 24339 ts_tablet_manager.cc:1397] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.004s
I20250626 02:03:04.347332 24339 raft_consensus.cc:357] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:04.347515 24339 raft_consensus.cc:738] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46dfe87116554377a3d4bf78c9f9699b, State: Initialized, Role: LEARNER
I20250626 02:03:04.347669 24339 consensus_queue.cc:260] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:04.347961 24341 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:04.348066 24339 ts_tablet_manager.cc:1428] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:04.348316 24341 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap complete.
I20250626 02:03:04.348527 24341 ts_tablet_manager.cc:1397] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: Time spent bootstrapping tablet: real 0.039s user 0.023s sys 0.003s
I20250626 02:03:04.348863 23903 tablet_copy_service.cc:342] P d150470bce35446b9b3f41ec78cb5064: Request end of tablet copy session 46dfe87116554377a3d4bf78c9f9699b-34ac2721a9bb44efa815122997c53fd7 received from {username='slave'} at 127.0.0.1:37572
I20250626 02:03:04.348948 23903 tablet_copy_service.cc:434] P d150470bce35446b9b3f41ec78cb5064: ending tablet copy session 46dfe87116554377a3d4bf78c9f9699b-34ac2721a9bb44efa815122997c53fd7 on tablet 34ac2721a9bb44efa815122997c53fd7 with peer 46dfe87116554377a3d4bf78c9f9699b
I20250626 02:03:04.349407 24341 raft_consensus.cc:357] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:04.349568 24341 raft_consensus.cc:738] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46dfe87116554377a3d4bf78c9f9699b, State: Initialized, Role: LEARNER
I20250626 02:03:04.349716 24341 consensus_queue.cc:260] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:04.350067 24341 ts_tablet_manager.cc:1428] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:04.350791 23752 tablet_copy_service.cc:342] P 3a8897880da64721ad84d66ff1fcf3a4: Request end of tablet copy session 46dfe87116554377a3d4bf78c9f9699b-3a4a1bceeb5a4e3faba6c6ee9f5ec534 received from {username='slave'} at 127.0.0.1:40636
I20250626 02:03:04.350996 23752 tablet_copy_service.cc:434] P 3a8897880da64721ad84d66ff1fcf3a4: ending tablet copy session 46dfe87116554377a3d4bf78c9f9699b-3a4a1bceeb5a4e3faba6c6ee9f5ec534 on tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with peer 46dfe87116554377a3d4bf78c9f9699b
I20250626 02:03:04.355334 24346 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=89 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:04.355624 24346 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap complete.
I20250626 02:03:04.355824 24346 ts_tablet_manager.cc:1397] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.000s
I20250626 02:03:04.356261 24346 raft_consensus.cc:357] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:04.356360 24346 raft_consensus.cc:738] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 145a19f8089541e3845de3dc9a3ac9dd, State: Initialized, Role: LEARNER
I20250626 02:03:04.356444 24346 consensus_queue.cc:260] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:04.356717 24346 ts_tablet_manager.cc:1428] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250626 02:03:04.357160 23903 tablet_copy_service.cc:342] P d150470bce35446b9b3f41ec78cb5064: Request end of tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-e37650dfb570400fb02742cb1220c7e1 received from {username='slave'} at 127.0.0.1:37576
I20250626 02:03:04.357235 23903 tablet_copy_service.cc:434] P d150470bce35446b9b3f41ec78cb5064: ending tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-e37650dfb570400fb02742cb1220c7e1 on tablet e37650dfb570400fb02742cb1220c7e1 with peer 145a19f8089541e3845de3dc9a3ac9dd
I20250626 02:03:04.470374 24348 tablet_copy_client.cc:538] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:04.471721 24348 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap starting.
I20250626 02:03:04.496002 24348 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:04.496359 24348 tablet_bootstrap.cc:492] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap complete.
I20250626 02:03:04.496577 24348 ts_tablet_manager.cc:1397] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.004s
I20250626 02:03:04.497059 24348 raft_consensus.cc:357] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:04.497180 24348 raft_consensus.cc:738] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5ee348337b146838a15e2d7161f5fb8, State: Initialized, Role: LEARNER
I20250626 02:03:04.497298 24348 consensus_queue.cc:260] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:04.497597 24348 ts_tablet_manager.cc:1428] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:04.498034 23828 tablet_copy_service.cc:342] P 6c2126b790c0477595d5b94c658b8a80: Request end of tablet copy session c5ee348337b146838a15e2d7161f5fb8-65bf2a25991b4d4a94efbc900569a162 received from {username='slave'} at 127.0.0.1:52060
I20250626 02:03:04.498111 23828 tablet_copy_service.cc:434] P 6c2126b790c0477595d5b94c658b8a80: ending tablet copy session c5ee348337b146838a15e2d7161f5fb8-65bf2a25991b4d4a94efbc900569a162 on tablet 65bf2a25991b4d4a94efbc900569a162 with peer c5ee348337b146838a15e2d7161f5fb8
I20250626 02:03:04.670552 24283 raft_consensus.cc:1215] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250626 02:03:04.703296 24132 raft_consensus.cc:1215] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20250626 02:03:04.726281 24354 raft_consensus.cc:1062] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: attempting to promote NON_VOTER c5ee348337b146838a15e2d7161f5fb8 to VOTER
I20250626 02:03:04.726755 24354 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:04.727943 24208 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:04.728101 23818 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:04.728209 23892 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:04.728320 24354 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:04.729349 24369 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20250626 02:03:04.729578 24366 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:04.731240 24368 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:04.731848 23818 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:04.732259 23892 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:04.733884 24208 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:04.734736 23633 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:04.738853 23739 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:04.739933 24208 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 02:03:04.740067 23818 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 02:03:04.740392 24369 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:04.740583 24368 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:04.742097 24368 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:04.742651 23818 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:04.743646 23626 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 363d650f886244d38b78e6af7bb5f6ce with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:04.744283 24208 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:04.744661 23635 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index 30 to 31, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:04.747210 23883 tablet_service.cc:1515] Processing DeleteTablet for tablet 363d650f886244d38b78e6af7bb5f6ce with delete_type TABLET_DATA_TOMBSTONED (TS d150470bce35446b9b3f41ec78cb5064 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:04.748171 24377 tablet_replica.cc:331] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:04.748315 24377 raft_consensus.cc:2241] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:04.748549 24377 raft_consensus.cc:2270] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:04.749090 24377 ts_tablet_manager.cc:1905] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:04.751248 24377 ts_tablet_manager.cc:1918] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20250626 02:03:04.751318 24377 log.cc:1199] T 363d650f886244d38b78e6af7bb5f6ce P d150470bce35446b9b3f41ec78cb5064: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/363d650f886244d38b78e6af7bb5f6ce
I20250626 02:03:04.751776 23623 catalog_manager.cc:4928] TS d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133): tablet 363d650f886244d38b78e6af7bb5f6ce (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:04.790371 24283 raft_consensus.cc:1215] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250626 02:03:04.845208 24132 raft_consensus.cc:1215] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250626 02:03:04.863762 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:04.872113 23891 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:47872
I20250626 02:03:04.872295 23891 raft_consensus.cc:604] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:04.876195 23739 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:04.876345 23739 raft_consensus.cc:604] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Received request to transfer leadership to TS 6c2126b790c0477595d5b94c658b8a80
I20250626 02:03:04.880257 23817 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:04.880395 23817 raft_consensus.cc:604] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:04.880718 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 02:03:04.880858 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:04.902398 24368 raft_consensus.cc:991] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:04.902530 24368 raft_consensus.cc:1079] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:04.902917 23817 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:51998
I20250626 02:03:04.903053 23817 raft_consensus.cc:491] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:04.903120 23817 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:04.904102 23817 raft_consensus.cc:513] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:04.904543 23817 leader_election.cc:290] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 2 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:04.904742 23739 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:04.904850 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:04.904959 23892 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:04.904856 23739 raft_consensus.cc:3053] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:04.905041 23739 raft_consensus.cc:738] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:04.905176 23739 consensus_queue.cc:260] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:04.905297 23739 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:04.905694 23892 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 2.
I20250626 02:03:04.906001 23739 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 2.
I20250626 02:03:04.905997 23781 leader_election.cc:304] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:04.906255 24352 raft_consensus.cc:2802] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:04.906396 24352 raft_consensus.cc:695] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:04.906490 24352 consensus_queue.cc:237] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:04.907929 23634 catalog_manager.cc:5582] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 1 to 2, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 2 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:04.963066 24208 raft_consensus.cc:1215] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20250626 02:03:05.091578 24325 raft_consensus.cc:1062] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: attempting to promote NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd to VOTER
I20250626 02:03:05.091964 24325 consensus_queue.cc:237] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:05.093297 23818 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 02:03:05.093315 24132 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 02:03:05.093298 23739 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 02:03:05.093803 24384 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 02:03:05.094136 24386 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 02:03:05.094305 24325 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 02:03:05.096418 24386 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.096796 23818 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.097167 23739 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.099454 23634 catalog_manager.cc:5582] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index 26 to 27, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.099592 24132 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.103129 23892 consensus_queue.cc:237] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:05.104223 24132 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 02:03:05.104302 23739 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 02:03:05.104650 24325 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:05.104833 24365 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:05.106231 24365 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.106729 24132 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.106789 23739 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.107761 23623 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e37650dfb570400fb02742cb1220c7e1 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:05.108590 23635 catalog_manager.cc:5582] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd reported cstate change: config changed from index 27 to 28, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.110782 23807 tablet_service.cc:1515] Processing DeleteTablet for tablet e37650dfb570400fb02742cb1220c7e1 with delete_type TABLET_DATA_TOMBSTONED (TS 6c2126b790c0477595d5b94c658b8a80 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:51988
I20250626 02:03:05.111202 24390 tablet_replica.cc:331] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:05.111351 24390 raft_consensus.cc:2241] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:05.111497 24390 raft_consensus.cc:2270] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:05.112046 24390 ts_tablet_manager.cc:1905] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:05.113968 24390 ts_tablet_manager.cc:1918] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20250626 02:03:05.114100 24390 log.cc:1199] T e37650dfb570400fb02742cb1220c7e1 P 6c2126b790c0477595d5b94c658b8a80: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/e37650dfb570400fb02742cb1220c7e1
I20250626 02:03:05.114562 23625 catalog_manager.cc:4928] TS 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183): tablet e37650dfb570400fb02742cb1220c7e1 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:05.135370 24382 raft_consensus.cc:991] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:05.135547 24382 raft_consensus.cc:1079] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:05.135938 23739 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40588
I20250626 02:03:05.136076 23739 raft_consensus.cc:491] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:05.136173 23739 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:05.136969 23739 raft_consensus.cc:513] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:05.137270 23739 leader_election.cc:290] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 2 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:05.137508 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:05.137625 23818 raft_consensus.cc:3053] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:05.137665 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:05.137710 23818 raft_consensus.cc:738] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:05.137765 23892 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:05.137871 23818 consensus_queue.cc:260] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:05.138046 23818 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:05.138864 23892 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 2.
I20250626 02:03:05.138881 23818 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 2.
I20250626 02:03:05.139189 23705 leader_election.cc:304] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:05.139408 24368 raft_consensus.cc:2802] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:05.139501 24368 raft_consensus.cc:695] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:05.139622 24368 consensus_queue.cc:237] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:05.141079 23633 catalog_manager.cc:5582] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 1 to 2, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 2 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:05.144044 24384 raft_consensus.cc:991] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:05.144170 24384 raft_consensus.cc:1079] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:05.144524 23739 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40596
I20250626 02:03:05.144616 23739 raft_consensus.cc:491] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:05.144658 23739 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:05.145583 23739 raft_consensus.cc:513] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:05.145944 23739 leader_election.cc:290] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 2 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:05.146173 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:05.146220 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:05.146301 23818 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:05.146350 23892 raft_consensus.cc:3053] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:05.146436 23892 raft_consensus.cc:738] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:05.146577 23892 consensus_queue.cc:260] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:05.146724 23892 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:05.147135 23818 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 2.
I20250626 02:03:05.147424 23707 leader_election.cc:304] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 6c2126b790c0477595d5b94c658b8a80; no voters:
I20250626 02:03:05.147572 24368 raft_consensus.cc:2802] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:05.147663 24368 raft_consensus.cc:695] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:05.147785 24368 consensus_queue.cc:237] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:05.147504 23892 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 2.
I20250626 02:03:05.149330 23633 catalog_manager.cc:5582] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 1 to 2, leader changed from d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 2 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:05.222659 24354 raft_consensus.cc:1062] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: attempting to promote NON_VOTER 46dfe87116554377a3d4bf78c9f9699b to VOTER
I20250626 02:03:05.223140 24354 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:05.224510 23891 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.224535 23817 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.224535 24283 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 LEARNER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.224866 24366 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.225145 24368 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.225289 24366 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.226899 24366 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.227308 23817 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.229321 23633 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.229403 23891 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.230449 24283 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.234653 23739 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:05.235631 23817 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.235886 24283 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.235983 24366 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.236287 24354 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.237322 24366 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.238263 24283 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.238785 23626 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:05.238973 23817 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.239746 23634 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index 29 to 30, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:05.242194 23883 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with delete_type TABLET_DATA_TOMBSTONED (TS d150470bce35446b9b3f41ec78cb5064 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:05.242385 24377 tablet_replica.cc:331] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:05.242525 24377 raft_consensus.cc:2241] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:05.242667 24377 raft_consensus.cc:2270] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:05.243254 24377 ts_tablet_manager.cc:1905] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:05.244951 24377 ts_tablet_manager.cc:1918] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250626 02:03:05.245030 24377 log.cc:1199] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P d150470bce35446b9b3f41ec78cb5064: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/3a4a1bceeb5a4e3faba6c6ee9f5ec534
I20250626 02:03:05.245468 23623 catalog_manager.cc:4928] TS d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133): tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:05.315618 24325 raft_consensus.cc:1062] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: attempting to promote NON_VOTER 46dfe87116554377a3d4bf78c9f9699b to VOTER
I20250626 02:03:05.316059 24325 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:05.317382 23739 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.317382 23817 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.317759 24325 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.318073 24325 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.318511 24283 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 LEARNER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.318838 24386 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.320300 24384 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.320761 23817 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.320881 24283 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.322654 23739 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.323338 23635 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 28 to 29, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:05.326566 23891 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:05.327606 24283 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.327713 23739 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.327961 24325 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.328171 24365 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.329540 24386 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.330013 23739 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.330049 24283 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:05.331673 23639 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 29 to 30, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:05.331990 23623 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 34ac2721a9bb44efa815122997c53fd7 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:05.334661 23807 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ac2721a9bb44efa815122997c53fd7 with delete_type TABLET_DATA_TOMBSTONED (TS 6c2126b790c0477595d5b94c658b8a80 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:51988
I20250626 02:03:05.334860 24390 tablet_replica.cc:331] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:05.334968 24390 raft_consensus.cc:2241] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:05.335094 24390 raft_consensus.cc:2270] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:05.335597 24390 ts_tablet_manager.cc:1905] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:05.337585 24390 ts_tablet_manager.cc:1918] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250626 02:03:05.337641 24390 log.cc:1199] T 34ac2721a9bb44efa815122997c53fd7 P 6c2126b790c0477595d5b94c658b8a80: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/34ac2721a9bb44efa815122997c53fd7
I20250626 02:03:05.338061 23625 catalog_manager.cc:4928] TS 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183): tablet 34ac2721a9bb44efa815122997c53fd7 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:05.365267 24352 raft_consensus.cc:1062] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: attempting to promote NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd to VOTER
I20250626 02:03:05.365661 24352 consensus_queue.cc:237] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:05.366822 24132 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.366822 23892 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.366950 23739 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:05.367311 24403 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.367496 24382 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.367710 24401 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:05.370025 24382 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.370504 23892 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.371829 24132 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.372875 23634 catalog_manager.cc:5582] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index 28 to 29, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:05.373687 23739 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.375809 23818 consensus_queue.cc:237] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:05.377436 24132 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.377924 24401 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.378073 23892 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.378378 24401 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.379503 24403 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.379822 24132 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.380216 23892 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.381430 23625 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 894851ab3c1e4d79971597e62b99cfbf with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:05.381979 23633 catalog_manager.cc:5582] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd reported cstate change: config changed from index 29 to 30, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:05.384229 23727 tablet_service.cc:1515] Processing DeleteTablet for tablet 894851ab3c1e4d79971597e62b99cfbf with delete_type TABLET_DATA_TOMBSTONED (TS 3a8897880da64721ad84d66ff1fcf3a4 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:40586
I20250626 02:03:05.384605 24407 tablet_replica.cc:331] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:05.384743 24407 raft_consensus.cc:2241] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:05.384948 24407 raft_consensus.cc:2270] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:05.385515 24407 ts_tablet_manager.cc:1905] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:05.387585 24407 ts_tablet_manager.cc:1918] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250626 02:03:05.387665 24407 log.cc:1199] T 894851ab3c1e4d79971597e62b99cfbf P 3a8897880da64721ad84d66ff1fcf3a4: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/894851ab3c1e4d79971597e62b99cfbf
I20250626 02:03:05.388182 23626 catalog_manager.cc:4928] TS 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725): tablet 894851ab3c1e4d79971597e62b99cfbf (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:05.419386 23739 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 02:03:05.419759 24401 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:05.422052 23892 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 02:03:05.422463 24401 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:05.447958 24352 raft_consensus.cc:1062] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: attempting to promote NON_VOTER c5ee348337b146838a15e2d7161f5fb8 to VOTER
I20250626 02:03:05.448498 24352 consensus_queue.cc:237] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:05.449743 23739 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.450006 24208 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.450006 23892 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:05.450205 24401 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.450383 24382 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.450582 24352 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:05.452402 24403 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:05.452930 24208 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:05.452939 23739 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:05.454420 23892 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:05.455389 23639 catalog_manager.cc:5582] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:05.458775 23818 consensus_queue.cc:237] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:05.459683 23892 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 02:03:05.459769 24208 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 02:03:05.460044 24352 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:05.460247 24382 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:05.461916 24401 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:05.462270 24208 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:05.462313 23892 raft_consensus.cc:2953] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:05.463634 23625 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 65bf2a25991b4d4a94efbc900569a162 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:05.464282 23634 catalog_manager.cc:5582] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: config changed from index 30 to 31, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:05.466423 23727 tablet_service.cc:1515] Processing DeleteTablet for tablet 65bf2a25991b4d4a94efbc900569a162 with delete_type TABLET_DATA_TOMBSTONED (TS 3a8897880da64721ad84d66ff1fcf3a4 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:40586
I20250626 02:03:05.466606 24407 tablet_replica.cc:331] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:05.466732 24407 raft_consensus.cc:2241] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:05.466889 24407 raft_consensus.cc:2270] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:05.467478 24407 ts_tablet_manager.cc:1905] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:05.469662 24407 ts_tablet_manager.cc:1918] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20250626 02:03:05.469746 24407 log.cc:1199] T 65bf2a25991b4d4a94efbc900569a162 P 3a8897880da64721ad84d66ff1fcf3a4: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/65bf2a25991b4d4a94efbc900569a162
I20250626 02:03:05.470297 23626 catalog_manager.cc:4928] TS 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725): tablet 65bf2a25991b4d4a94efbc900569a162 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:05.566375 23892 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 02:03:05.566799 24366 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:05.569113 23818 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 02:03:05.569504 24368 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:05.634280 23892 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 02:03:05.634819 24368 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:05.640587 23818 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 02:03:05.641283 24354 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 02:03:06.468518 23739 consensus_queue.cc:237] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:06.469769 23892 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 02:03:06.469836 23818 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 02:03:06.470095 24422 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:06.470351 24421 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:06.472286 24421 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:06.473273 23892 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:06.474851 23818 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:06.475180 23891 consensus_queue.cc:237] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:06.475178 23633 catalog_manager.cc:5582] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index -1 to 29, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New cstate: current_term: 2 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 02:03:06.476569 23708 consensus_peers.cc:487] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e38ec4282a4b40e8bb9d07ef486659d6. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:06.477380 24132 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:06.477566 23739 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:06.477751 24429 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:06.477924 24427 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:06.479509 24429 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:06.479851 23739 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:06.482228 24132 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
W20250626 02:03:06.482694 23857 consensus_peers.cc:487] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 -> Peer c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947): Couldn't send request to peer c5ee348337b146838a15e2d7161f5fb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e37650dfb570400fb02742cb1220c7e1. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:06.483923 23634 catalog_manager.cc:5582] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 28 to 29, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:06.483927 23739 consensus_queue.cc:237] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:06.485167 23891 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
W20250626 02:03:06.485231 23705 consensus_peers.cc:487] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 -> Peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101): Couldn't send request to peer 46dfe87116554377a3d4bf78c9f9699b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13501c41bba94a7bbfb48a2715cd2d40. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:06.485546 23818 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 02:03:06.485816 24354 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:06.485971 24422 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:06.487174 24354 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:06.487767 23891 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:06.487908 23818 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:06.490000 23635 catalog_manager.cc:5582] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index -1 to 29, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New cstate: current_term: 2 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:06.490564 23818 consensus_queue.cc:237] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:06.492026 23739 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:06.492026 23891 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 02:03:06.492926 24442 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:06.493086 24441 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:06.494673 24441 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:06.495060 23739 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:06.496827 23891 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
W20250626 02:03:06.497299 23781 consensus_peers.cc:487] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 -> Peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101): Couldn't send request to peer 46dfe87116554377a3d4bf78c9f9699b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70698cc805a448b28cef88e47e146427. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:06.497694 23634 catalog_manager.cc:5582] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index -1 to 29, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New cstate: current_term: 1 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:06.499194 23891 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:06.500691 24283 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 02:03:06.500701 23739 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
W20250626 02:03:06.500857 23857 consensus_peers.cc:487] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 -> Peer c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947): Couldn't send request to peer c5ee348337b146838a15e2d7161f5fb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34ac2721a9bb44efa815122997c53fd7. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:06.501030 24437 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:06.501191 24427 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:06.502758 24429 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:06.503160 24283 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:06.503211 23739 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:06.505187 23635 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 30 to 31, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:06.505535 23739 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:06.507107 24208 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
W20250626 02:03:06.507349 23708 consensus_peers.cc:487] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 363d650f886244d38b78e6af7bb5f6ce. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:06.507362 23818 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250626 02:03:06.507555 24422 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:06.507802 24354 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:06.509191 24421 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:06.509557 24208 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:06.511145 23634 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index 31 to 32, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:06.511353 23818 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:06.855573 24456 ts_tablet_manager.cc:927] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: Initiating tablet copy from peer 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725)
I20250626 02:03:06.856067 24456 tablet_copy_client.cc:323] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:34725
I20250626 02:03:06.859911 23752 tablet_copy_service.cc:140] P 3a8897880da64721ad84d66ff1fcf3a4: Received BeginTabletCopySession request for tablet e38ec4282a4b40e8bb9d07ef486659d6 from peer 145a19f8089541e3845de3dc9a3ac9dd ({username='slave'} at 127.0.0.1:32946)
I20250626 02:03:06.860005 23752 tablet_copy_service.cc:161] P 3a8897880da64721ad84d66ff1fcf3a4: Beginning new tablet copy session on tablet e38ec4282a4b40e8bb9d07ef486659d6 from peer 145a19f8089541e3845de3dc9a3ac9dd at {username='slave'} at 127.0.0.1:32946: session id = 145a19f8089541e3845de3dc9a3ac9dd-e38ec4282a4b40e8bb9d07ef486659d6
I20250626 02:03:06.861001 23752 tablet_copy_source_session.cc:215] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:06.861783 24456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38ec4282a4b40e8bb9d07ef486659d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:06.864555 24456 tablet_copy_client.cc:806] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:06.864753 24456 tablet_copy_client.cc:670] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:06.865650 24457 tablet_copy_client.cc:1025] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:06.881073 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:06.882861 23739 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "363d650f886244d38b78e6af7bb5f6ce"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "c5ee348337b146838a15e2d7161f5fb8"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:06.883002 23739 raft_consensus.cc:604] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Received request to transfer leadership to TS c5ee348337b146838a15e2d7161f5fb8
I20250626 02:03:06.883730 23739 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3a4a1bceeb5a4e3faba6c6ee9f5ec534"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "46dfe87116554377a3d4bf78c9f9699b"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:06.883834 23739 raft_consensus.cc:604] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Received request to transfer leadership to TS 46dfe87116554377a3d4bf78c9f9699b
I20250626 02:03:06.884495 23818 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "70698cc805a448b28cef88e47e146427"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:06.884613 23818 raft_consensus.cc:604] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:06.885291 23818 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "894851ab3c1e4d79971597e62b99cfbf"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "145a19f8089541e3845de3dc9a3ac9dd"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:06.885401 23818 raft_consensus.cc:604] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Received request to transfer leadership to TS 145a19f8089541e3845de3dc9a3ac9dd
I20250626 02:03:06.886139 23891 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
mode: GRACEFUL
new_leader_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:47872
I20250626 02:03:06.886255 23891 raft_consensus.cc:604] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Received request to transfer leadership to TS 6c2126b790c0477595d5b94c658b8a80
I20250626 02:03:06.886885 23739 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "d150470bce35446b9b3f41ec78cb5064"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:06.887001 23739 raft_consensus.cc:604] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Received request to transfer leadership to TS d150470bce35446b9b3f41ec78cb5064
I20250626 02:03:06.887264 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250626 02:03:06.887395 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:06.903266 24421 raft_consensus.cc:991] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower d150470bce35446b9b3f41ec78cb5064 to start an election
I20250626 02:03:06.903383 24421 raft_consensus.cc:1079] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Signalling peer d150470bce35446b9b3f41ec78cb5064 to start an election
I20250626 02:03:06.903775 23891 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
from {username='slave'} at 127.0.0.1:37532
I20250626 02:03:06.903914 23891 raft_consensus.cc:491] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:06.903985 23891 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:06.904856 23891 raft_consensus.cc:513] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:06.905203 23891 leader_election.cc:290] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 3 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:06.905511 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:06.905615 23818 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:06.906446 23818 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 3.
I20250626 02:03:06.906654 23739 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:06.906737 23858 leader_election.cc:304] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:06.906795 23739 raft_consensus.cc:3053] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:03:06.906831 23739 raft_consensus.cc:738] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:06.906904 24437 raft_consensus.cc:2802] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:03:06.907025 24437 raft_consensus.cc:695] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Becoming Leader. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:06.906988 23739 consensus_queue.cc:260] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:06.907209 23739 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:06.907181 24437 consensus_queue.cc:237] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:06.907835 24460 ts_tablet_manager.cc:927] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: Initiating tablet copy from peer 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725)
I20250626 02:03:06.908758 23635 catalog_manager.cc:5582] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: term changed from 2 to 3, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to d150470bce35446b9b3f41ec78cb5064 (127.19.220.131). New cstate: current_term: 3 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:06.909054 23739 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 3.
I20250626 02:03:06.909523 24460 tablet_copy_client.cc:323] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:34725
I20250626 02:03:06.909920 23752 tablet_copy_service.cc:140] P 3a8897880da64721ad84d66ff1fcf3a4: Received BeginTabletCopySession request for tablet 363d650f886244d38b78e6af7bb5f6ce from peer 145a19f8089541e3845de3dc9a3ac9dd ({username='slave'} at 127.0.0.1:32946)
I20250626 02:03:06.910003 23752 tablet_copy_service.cc:161] P 3a8897880da64721ad84d66ff1fcf3a4: Beginning new tablet copy session on tablet 363d650f886244d38b78e6af7bb5f6ce from peer 145a19f8089541e3845de3dc9a3ac9dd at {username='slave'} at 127.0.0.1:32946: session id = 145a19f8089541e3845de3dc9a3ac9dd-363d650f886244d38b78e6af7bb5f6ce
I20250626 02:03:06.910915 23752 tablet_copy_source_session.cc:215] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:06.911667 24460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363d650f886244d38b78e6af7bb5f6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:06.912571 23891 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:06.912698 23891 raft_consensus.cc:604] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Received request to transfer leadership
I20250626 02:03:06.913172 24421 raft_consensus.cc:991] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower c5ee348337b146838a15e2d7161f5fb8 to start an election
I20250626 02:03:06.913270 24354 raft_consensus.cc:1079] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Signalling peer c5ee348337b146838a15e2d7161f5fb8 to start an election
I20250626 02:03:06.913690 24208 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "363d650f886244d38b78e6af7bb5f6ce"
dest_uuid: "c5ee348337b146838a15e2d7161f5fb8"
from {username='slave'} at 127.0.0.1:39320
I20250626 02:03:06.913784 24208 raft_consensus.cc:491] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:06.913832 24208 raft_consensus.cc:3058] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:06.913909 24460 tablet_copy_client.cc:806] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:06.914028 24460 tablet_copy_client.cc:670] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:06.914724 24208 raft_consensus.cc:513] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:06.915154 24208 leader_election.cc:290] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [CANDIDATE]: Term 2 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725)
I20250626 02:03:06.915350 24461 tablet_copy_client.cc:1025] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:06.915407 23739 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "363d650f886244d38b78e6af7bb5f6ce" candidate_uuid: "c5ee348337b146838a15e2d7161f5fb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:06.915510 23739 raft_consensus.cc:3053] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:06.915557 23739 raft_consensus.cc:738] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:06.915530 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "363d650f886244d38b78e6af7bb5f6ce" candidate_uuid: "c5ee348337b146838a15e2d7161f5fb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:06.915674 23818 raft_consensus.cc:3058] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:06.915711 23739 consensus_queue.cc:260] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:06.915848 23739 raft_consensus.cc:3058] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:06.916616 23818 raft_consensus.cc:2466] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ee348337b146838a15e2d7161f5fb8 in term 2.
I20250626 02:03:06.916898 24173 leader_election.cc:304] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, c5ee348337b146838a15e2d7161f5fb8; no voters:
I20250626 02:03:06.916908 23739 raft_consensus.cc:2466] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ee348337b146838a15e2d7161f5fb8 in term 2.
I20250626 02:03:06.917117 24455 raft_consensus.cc:2802] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:06.917326 24455 raft_consensus.cc:695] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEADER]: Becoming Leader. State: Replica: c5ee348337b146838a15e2d7161f5fb8, State: Running, Role: LEADER
I20250626 02:03:06.917436 24455 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:06.918924 23635 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: term changed from 1 to 2, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133). New cstate: current_term: 2 leader_uuid: "c5ee348337b146838a15e2d7161f5fb8" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:06.919308 23891 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:06.919395 23891 raft_consensus.cc:604] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Received request to transfer leadership
W20250626 02:03:06.919703 23623 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for e38ec4282a4b40e8bb9d07ef486659d6 already in progress [suppressed 8 similar messages]
I20250626 02:03:06.920655 23891 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:06.920753 23891 raft_consensus.cc:604] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Received request to transfer leadership
W20250626 02:03:06.921067 23688 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for e38ec4282a4b40e8bb9d07ef486659d6 already in progress
W20250626 02:03:06.921128 23688 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for e38ec4282a4b40e8bb9d07ef486659d6 already in progress
I20250626 02:03:06.985703 24447 raft_consensus.cc:991] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:06.985831 24447 raft_consensus.cc:1079] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:06.986219 23740 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "70698cc805a448b28cef88e47e146427"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40588
I20250626 02:03:06.986352 23740 raft_consensus.cc:491] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:06.986436 23740 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:06.987450 23740 raft_consensus.cc:513] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:06.987831 23740 leader_election.cc:290] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 2 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:06.988004 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:06.988325 23891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:06.988355 23818 raft_consensus.cc:3053] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:06.988420 23891 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:06.988410 23818 raft_consensus.cc:738] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:06.988600 23818 consensus_queue.cc:260] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:06.988757 23818 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:06.989405 23891 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 2.
I20250626 02:03:06.989529 23818 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 2.
I20250626 02:03:06.989697 23705 leader_election.cc:304] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:06.989837 24354 raft_consensus.cc:2802] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:06.989928 24354 raft_consensus.cc:695] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:06.990041 24354 consensus_queue.cc:237] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:06.991464 23635 catalog_manager.cc:5582] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 1 to 2, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 2 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:07.044274 24465 ts_tablet_manager.cc:927] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: Initiating tablet copy from peer 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725)
I20250626 02:03:07.044641 24465 tablet_copy_client.cc:323] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:34725
I20250626 02:03:07.045060 23752 tablet_copy_service.cc:140] P 3a8897880da64721ad84d66ff1fcf3a4: Received BeginTabletCopySession request for tablet 13501c41bba94a7bbfb48a2715cd2d40 from peer 46dfe87116554377a3d4bf78c9f9699b ({username='slave'} at 127.0.0.1:40636)
I20250626 02:03:07.045161 23752 tablet_copy_service.cc:161] P 3a8897880da64721ad84d66ff1fcf3a4: Beginning new tablet copy session on tablet 13501c41bba94a7bbfb48a2715cd2d40 from peer 46dfe87116554377a3d4bf78c9f9699b at {username='slave'} at 127.0.0.1:40636: session id = 46dfe87116554377a3d4bf78c9f9699b-13501c41bba94a7bbfb48a2715cd2d40
I20250626 02:03:07.046181 23752 tablet_copy_source_session.cc:215] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:07.046195 24467 ts_tablet_manager.cc:927] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: Initiating tablet copy from peer d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:07.046707 24467 tablet_copy_client.cc:323] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:40133
I20250626 02:03:07.046958 24465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13501c41bba94a7bbfb48a2715cd2d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:07.049388 24465 tablet_copy_client.cc:806] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:07.049592 24465 tablet_copy_client.cc:670] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:07.050509 24466 tablet_copy_client.cc:1025] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:07.051831 23903 tablet_copy_service.cc:140] P d150470bce35446b9b3f41ec78cb5064: Received BeginTabletCopySession request for tablet e37650dfb570400fb02742cb1220c7e1 from peer c5ee348337b146838a15e2d7161f5fb8 ({username='slave'} at 127.0.0.1:47882)
I20250626 02:03:07.051932 23903 tablet_copy_service.cc:161] P d150470bce35446b9b3f41ec78cb5064: Beginning new tablet copy session on tablet e37650dfb570400fb02742cb1220c7e1 from peer c5ee348337b146838a15e2d7161f5fb8 at {username='slave'} at 127.0.0.1:47882: session id = c5ee348337b146838a15e2d7161f5fb8-e37650dfb570400fb02742cb1220c7e1
I20250626 02:03:07.052748 23903 tablet_copy_source_session.cc:215] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:07.053541 24467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e37650dfb570400fb02742cb1220c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:07.055730 24467 tablet_copy_client.cc:806] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:07.055886 24467 tablet_copy_client.cc:670] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:07.056918 24468 tablet_copy_client.cc:1025] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:07.085652 24471 ts_tablet_manager.cc:927] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: Initiating tablet copy from peer d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:07.086037 24471 tablet_copy_client.cc:323] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:40133
I20250626 02:03:07.086475 23903 tablet_copy_service.cc:140] P d150470bce35446b9b3f41ec78cb5064: Received BeginTabletCopySession request for tablet 34ac2721a9bb44efa815122997c53fd7 from peer c5ee348337b146838a15e2d7161f5fb8 ({username='slave'} at 127.0.0.1:47882)
I20250626 02:03:07.086568 23903 tablet_copy_service.cc:161] P d150470bce35446b9b3f41ec78cb5064: Beginning new tablet copy session on tablet 34ac2721a9bb44efa815122997c53fd7 from peer c5ee348337b146838a15e2d7161f5fb8 at {username='slave'} at 127.0.0.1:47882: session id = c5ee348337b146838a15e2d7161f5fb8-34ac2721a9bb44efa815122997c53fd7
I20250626 02:03:07.087389 23903 tablet_copy_source_session.cc:215] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:07.088156 24471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac2721a9bb44efa815122997c53fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:07.090267 24471 tablet_copy_client.cc:806] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:07.090430 24471 tablet_copy_client.cc:670] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:07.091444 24472 tablet_copy_client.cc:1025] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:07.253895 24421 raft_consensus.cc:991] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower 46dfe87116554377a3d4bf78c9f9699b to start an election
I20250626 02:03:07.254024 24421 raft_consensus.cc:1079] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Signalling peer 46dfe87116554377a3d4bf78c9f9699b to start an election
I20250626 02:03:07.254359 24283 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3a4a1bceeb5a4e3faba6c6ee9f5ec534"
dest_uuid: "46dfe87116554377a3d4bf78c9f9699b"
from {username='slave'} at 127.0.0.1:54574
I20250626 02:03:07.254501 24283 raft_consensus.cc:491] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:07.254571 24283 raft_consensus.cc:3058] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.255548 24283 raft_consensus.cc:513] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:07.256021 24283 leader_election.cc:290] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [CANDIDATE]: Term 2 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:07.256311 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a4a1bceeb5a4e3faba6c6ee9f5ec534" candidate_uuid: "46dfe87116554377a3d4bf78c9f9699b" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:07.256475 23740 raft_consensus.cc:3053] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:07.256619 23740 raft_consensus.cc:738] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:07.256811 23740 consensus_queue.cc:260] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:07.256953 23740 raft_consensus.cc:3058] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.258035 23740 raft_consensus.cc:2466] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46dfe87116554377a3d4bf78c9f9699b in term 2.
I20250626 02:03:07.258324 24249 leader_election.cc:304] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 46dfe87116554377a3d4bf78c9f9699b; no voters:
I20250626 02:03:07.258637 24475 raft_consensus.cc:2802] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:07.258875 24475 raft_consensus.cc:695] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Becoming Leader. State: Replica: 46dfe87116554377a3d4bf78c9f9699b, State: Running, Role: LEADER
I20250626 02:03:07.259294 24475 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:07.261212 23635 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: term changed from 1 to 2, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134). New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:07.262778 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a4a1bceeb5a4e3faba6c6ee9f5ec534" candidate_uuid: "46dfe87116554377a3d4bf78c9f9699b" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:07.262888 23818 raft_consensus.cc:3058] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.263876 23818 raft_consensus.cc:2466] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46dfe87116554377a3d4bf78c9f9699b in term 2.
I20250626 02:03:07.298852 24384 raft_consensus.cc:991] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:07.298985 24384 raft_consensus.cc:1079] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:07.299348 23818 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:52008
I20250626 02:03:07.299491 23818 raft_consensus.cc:491] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:07.299562 23818 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.300758 23818 raft_consensus.cc:513] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:07.301163 23818 leader_election.cc:290] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 2 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:07.301503 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:07.301589 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:07.301633 23892 raft_consensus.cc:3053] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:07.301676 23892 raft_consensus.cc:738] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:07.301683 23740 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.301784 23892 consensus_queue.cc:260] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:07.301903 23892 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.302451 23740 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 2.
I20250626 02:03:07.302628 23892 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 2.
I20250626 02:03:07.302690 23784 leader_election.cc:304] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 6c2126b790c0477595d5b94c658b8a80; no voters:
I20250626 02:03:07.302841 24447 raft_consensus.cc:2802] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:07.302922 24447 raft_consensus.cc:695] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:07.303033 24447 consensus_queue.cc:237] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:07.304620 23635 catalog_manager.cc:5582] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 1 to 2, leader changed from d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 2 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:07.326226 23818 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 02:03:07.326620 24455 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:07.330149 23740 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 02:03:07.330651 24455 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
W20250626 02:03:07.333843 24174 consensus_peers.cc:487] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:07.355046 24447 raft_consensus.cc:991] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower 145a19f8089541e3845de3dc9a3ac9dd to start an election
I20250626 02:03:07.355158 24447 raft_consensus.cc:1079] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Signalling peer 145a19f8089541e3845de3dc9a3ac9dd to start an election
I20250626 02:03:07.355543 24132 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "894851ab3c1e4d79971597e62b99cfbf"
dest_uuid: "145a19f8089541e3845de3dc9a3ac9dd"
from {username='slave'} at 127.0.0.1:53788
I20250626 02:03:07.355671 24132 raft_consensus.cc:491] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:07.355755 24132 raft_consensus.cc:3058] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.356837 24132 raft_consensus.cc:513] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:07.357267 24132 leader_election.cc:290] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [CANDIDATE]: Term 2 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:07.357492 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "894851ab3c1e4d79971597e62b99cfbf" candidate_uuid: "145a19f8089541e3845de3dc9a3ac9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:07.357602 23818 raft_consensus.cc:3053] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:07.357616 23891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "894851ab3c1e4d79971597e62b99cfbf" candidate_uuid: "145a19f8089541e3845de3dc9a3ac9dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:07.357646 23818 raft_consensus.cc:738] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:07.357694 23891 raft_consensus.cc:3058] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.357800 23818 consensus_queue.cc:260] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:07.357973 23818 raft_consensus.cc:3058] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:07.358651 23891 raft_consensus.cc:2466] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 145a19f8089541e3845de3dc9a3ac9dd in term 2.
I20250626 02:03:07.358842 23818 raft_consensus.cc:2466] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 145a19f8089541e3845de3dc9a3ac9dd in term 2.
I20250626 02:03:07.358961 24095 leader_election.cc:304] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145a19f8089541e3845de3dc9a3ac9dd, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:07.359303 24484 raft_consensus.cc:2802] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:07.359498 24484 raft_consensus.cc:695] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEADER]: Becoming Leader. State: Replica: 145a19f8089541e3845de3dc9a3ac9dd, State: Running, Role: LEADER
I20250626 02:03:07.359966 24484 consensus_queue.cc:237] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:07.361341 23633 catalog_manager.cc:5582] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd reported cstate change: term changed from 1 to 2, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132). New cstate: current_term: 2 leader_uuid: "145a19f8089541e3845de3dc9a3ac9dd" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:07.361513 23818 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 02:03:07.361915 24384 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:07.363138 24437 raft_consensus.cc:991] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:07.363236 24384 raft_consensus.cc:1079] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:07.364295 23739 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20250626 02:03:07.364696 23859 consensus_peers.cc:487] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:07.365160 23817 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:52008
I20250626 02:03:07.365345 23817 raft_consensus.cc:491] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:07.365401 23817 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:07.365896 24384 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:07.366358 23817 raft_consensus.cc:513] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:07.366707 23817 leader_election.cc:290] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 4 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:07.367712 23739 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:07.367806 23739 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:07.368054 23891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e38ec4282a4b40e8bb9d07ef486659d6" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:07.368412 23891 raft_consensus.cc:3053] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 02:03:07.368468 23891 raft_consensus.cc:738] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:07.368603 23891 consensus_queue.cc:260] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:07.368661 23739 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 4.
I20250626 02:03:07.368767 23891 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:07.368891 23784 leader_election.cc:304] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 6c2126b790c0477595d5b94c658b8a80; no voters:
I20250626 02:03:07.369041 24442 raft_consensus.cc:2802] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 02:03:07.369100 24442 raft_consensus.cc:695] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:07.369180 24442 consensus_queue.cc:237] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:07.369506 23891 raft_consensus.cc:2466] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 4.
I20250626 02:03:07.370420 23635 catalog_manager.cc:5582] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 3 to 4, leader changed from d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 4 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:07.383661 23891 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 02:03:07.383986 24354 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:07.386111 23817 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 02:03:07.386472 24489 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20250626 02:03:07.387283 23705 consensus_peers.cc:487] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 -> Peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101): Couldn't send request to peer 46dfe87116554377a3d4bf78c9f9699b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70698cc805a448b28cef88e47e146427. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:07.667601 23739 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:07.668020 24475 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:07.670782 23817 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:07.671186 24492 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:07.753438 23740 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250626 02:03:07.754189 24442 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20250626 02:03:07.756772 23784 consensus_peers.cc:487] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:07.756875 23892 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250626 02:03:07.757714 24447 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:07.846293 23892 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 02:03:07.846747 24447 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:07.850198 23740 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 02:03:07.850531 24494 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:07.854571 23817 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:07.854965 24484 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:07.857929 23892 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:07.858317 24485 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:07.892699 24500 ts_tablet_manager.cc:927] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: Initiating tablet copy from peer 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725)
I20250626 02:03:07.893054 24500 tablet_copy_client.cc:323] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:34725
I20250626 02:03:07.893483 23752 tablet_copy_service.cc:140] P 3a8897880da64721ad84d66ff1fcf3a4: Received BeginTabletCopySession request for tablet 70698cc805a448b28cef88e47e146427 from peer 46dfe87116554377a3d4bf78c9f9699b ({username='slave'} at 127.0.0.1:40636)
I20250626 02:03:07.893584 23752 tablet_copy_service.cc:161] P 3a8897880da64721ad84d66ff1fcf3a4: Beginning new tablet copy session on tablet 70698cc805a448b28cef88e47e146427 from peer 46dfe87116554377a3d4bf78c9f9699b at {username='slave'} at 127.0.0.1:40636: session id = 46dfe87116554377a3d4bf78c9f9699b-70698cc805a448b28cef88e47e146427
I20250626 02:03:07.894376 23752 tablet_copy_source_session.cc:215] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:07.895143 24500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70698cc805a448b28cef88e47e146427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:07.897465 24500 tablet_copy_client.cc:806] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:07.897653 24500 tablet_copy_client.cc:670] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:07.898630 24501 tablet_copy_client.cc:1025] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:08.865931 24456 tablet_copy_client.cc:538] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:08.867679 24456 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap starting.
I20250626 02:03:08.887666 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:08.889719 24456 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:08.889819 23740 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "d150470bce35446b9b3f41ec78cb5064"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:08.889961 23740 raft_consensus.cc:604] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Received request to transfer leadership to TS d150470bce35446b9b3f41ec78cb5064
I20250626 02:03:08.890303 24456 tablet_bootstrap.cc:492] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap complete.
I20250626 02:03:08.890532 24456 ts_tablet_manager.cc:1397] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.007s
I20250626 02:03:08.890758 23817 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "d150470bce35446b9b3f41ec78cb5064"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:08.890864 23817 raft_consensus.cc:604] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Received request to transfer leadership to TS d150470bce35446b9b3f41ec78cb5064
I20250626 02:03:08.891041 24456 raft_consensus.cc:357] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:08.891180 24456 raft_consensus.cc:738] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 145a19f8089541e3845de3dc9a3ac9dd, State: Initialized, Role: LEARNER
I20250626 02:03:08.891304 24456 consensus_queue.cc:260] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:08.891532 23817 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:08.891623 23817 raft_consensus.cc:604] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:08.891654 24456 ts_tablet_manager.cc:1428] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250626 02:03:08.892030 23752 tablet_copy_service.cc:342] P 3a8897880da64721ad84d66ff1fcf3a4: Request end of tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-e38ec4282a4b40e8bb9d07ef486659d6 received from {username='slave'} at 127.0.0.1:32946
I20250626 02:03:08.892073 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 02:03:08.892103 23752 tablet_copy_service.cc:434] P 3a8897880da64721ad84d66ff1fcf3a4: ending tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-e38ec4282a4b40e8bb9d07ef486659d6 on tablet e38ec4282a4b40e8bb9d07ef486659d6 with peer 145a19f8089541e3845de3dc9a3ac9dd
I20250626 02:03:08.892200 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:08.915616 24460 tablet_copy_client.cc:538] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:08.918431 24460 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap starting.
I20250626 02:03:08.945228 24460 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:08.945557 24460 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap complete.
I20250626 02:03:08.945781 24460 ts_tablet_manager.cc:1397] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: Time spent bootstrapping tablet: real 0.027s user 0.026s sys 0.000s
I20250626 02:03:08.946256 24460 raft_consensus.cc:357] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:08.946377 24460 raft_consensus.cc:738] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 145a19f8089541e3845de3dc9a3ac9dd, State: Initialized, Role: LEARNER
I20250626 02:03:08.946497 24460 consensus_queue.cc:260] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:08.946839 24460 ts_tablet_manager.cc:1428] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:08.947280 23752 tablet_copy_service.cc:342] P 3a8897880da64721ad84d66ff1fcf3a4: Request end of tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-363d650f886244d38b78e6af7bb5f6ce received from {username='slave'} at 127.0.0.1:32946
I20250626 02:03:08.947360 23752 tablet_copy_service.cc:434] P 3a8897880da64721ad84d66ff1fcf3a4: ending tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-363d650f886244d38b78e6af7bb5f6ce on tablet 363d650f886244d38b78e6af7bb5f6ce with peer 145a19f8089541e3845de3dc9a3ac9dd
I20250626 02:03:09.044782 24132 raft_consensus.cc:3058] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 1 LEARNER]: Advancing to term 2
I20250626 02:03:09.050804 24465 tablet_copy_client.cc:538] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:09.052465 24465 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap starting.
I20250626 02:03:09.057188 24467 tablet_copy_client.cc:538] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:09.058619 24467 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap starting.
I20250626 02:03:09.075083 24465 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:09.075423 24465 tablet_bootstrap.cc:492] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap complete.
I20250626 02:03:09.075634 24465 ts_tablet_manager.cc:1397] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.004s
I20250626 02:03:09.076473 24465 raft_consensus.cc:357] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:09.076845 24465 raft_consensus.cc:738] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 46dfe87116554377a3d4bf78c9f9699b, State: Initialized, Role: LEARNER
I20250626 02:03:09.077029 24465 consensus_queue.cc:260] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:09.077538 24465 ts_tablet_manager.cc:1428] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250626 02:03:09.078334 23752 tablet_copy_service.cc:342] P 3a8897880da64721ad84d66ff1fcf3a4: Request end of tablet copy session 46dfe87116554377a3d4bf78c9f9699b-13501c41bba94a7bbfb48a2715cd2d40 received from {username='slave'} at 127.0.0.1:40636
I20250626 02:03:09.078410 23752 tablet_copy_service.cc:434] P 3a8897880da64721ad84d66ff1fcf3a4: ending tablet copy session 46dfe87116554377a3d4bf78c9f9699b-13501c41bba94a7bbfb48a2715cd2d40 on tablet 13501c41bba94a7bbfb48a2715cd2d40 with peer 46dfe87116554377a3d4bf78c9f9699b
I20250626 02:03:09.081681 24467 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=89 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:09.081959 24467 tablet_bootstrap.cc:492] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap complete.
I20250626 02:03:09.082149 24467 ts_tablet_manager.cc:1397] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20250626 02:03:09.082527 24467 raft_consensus.cc:357] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:09.082623 24467 raft_consensus.cc:738] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5ee348337b146838a15e2d7161f5fb8, State: Initialized, Role: LEARNER
I20250626 02:03:09.082751 24467 consensus_queue.cc:260] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:09.083043 24467 ts_tablet_manager.cc:1428] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250626 02:03:09.083467 23903 tablet_copy_service.cc:342] P d150470bce35446b9b3f41ec78cb5064: Request end of tablet copy session c5ee348337b146838a15e2d7161f5fb8-e37650dfb570400fb02742cb1220c7e1 received from {username='slave'} at 127.0.0.1:47882
I20250626 02:03:09.083551 23903 tablet_copy_service.cc:434] P d150470bce35446b9b3f41ec78cb5064: ending tablet copy session c5ee348337b146838a15e2d7161f5fb8-e37650dfb570400fb02742cb1220c7e1 on tablet e37650dfb570400fb02742cb1220c7e1 with peer c5ee348337b146838a15e2d7161f5fb8
I20250626 02:03:09.091727 24471 tablet_copy_client.cc:538] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:09.093341 24471 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap starting.
I20250626 02:03:09.100485 24494 raft_consensus.cc:991] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower d150470bce35446b9b3f41ec78cb5064 to start an election
I20250626 02:03:09.100574 24494 raft_consensus.cc:1079] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Signalling peer d150470bce35446b9b3f41ec78cb5064 to start an election
I20250626 02:03:09.100952 23892 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
from {username='slave'} at 127.0.0.1:37546
I20250626 02:03:09.101089 23892 raft_consensus.cc:491] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:09.101152 23892 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:09.101953 23892 raft_consensus.cc:513] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:09.102286 23892 leader_election.cc:290] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 2 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947)
I20250626 02:03:09.102547 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:09.102577 24207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "c5ee348337b146838a15e2d7161f5fb8"
I20250626 02:03:09.102649 23817 raft_consensus.cc:3053] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:09.102689 24207 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:09.102694 23817 raft_consensus.cc:738] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:09.102946 23817 consensus_queue.cc:260] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:09.103106 23817 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:09.103866 24207 raft_consensus.cc:2466] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 2.
I20250626 02:03:09.104173 23857 leader_election.cc:304] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ee348337b146838a15e2d7161f5fb8, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:09.104357 24502 raft_consensus.cc:2802] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:09.104530 24502 raft_consensus.cc:695] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 2 LEADER]: Becoming Leader. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:09.104657 24502 consensus_queue.cc:237] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:09.104081 23817 raft_consensus.cc:2466] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 2.
I20250626 02:03:09.106160 23635 catalog_manager.cc:5582] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: term changed from 1 to 2, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to d150470bce35446b9b3f41ec78cb5064 (127.19.220.131). New cstate: current_term: 2 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:09.118798 24471 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:09.119117 24471 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap complete.
I20250626 02:03:09.119304 24471 ts_tablet_manager.cc:1397] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20250626 02:03:09.119793 24471 raft_consensus.cc:357] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:09.119905 24471 raft_consensus.cc:738] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5ee348337b146838a15e2d7161f5fb8, State: Initialized, Role: LEARNER
I20250626 02:03:09.119982 24471 consensus_queue.cc:260] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:09.120287 24471 ts_tablet_manager.cc:1428] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250626 02:03:09.120756 23903 tablet_copy_service.cc:342] P d150470bce35446b9b3f41ec78cb5064: Request end of tablet copy session c5ee348337b146838a15e2d7161f5fb8-34ac2721a9bb44efa815122997c53fd7 received from {username='slave'} at 127.0.0.1:47882
I20250626 02:03:09.120842 23903 tablet_copy_service.cc:434] P d150470bce35446b9b3f41ec78cb5064: ending tablet copy session c5ee348337b146838a15e2d7161f5fb8-34ac2721a9bb44efa815122997c53fd7 on tablet 34ac2721a9bb44efa815122997c53fd7 with peer c5ee348337b146838a15e2d7161f5fb8
I20250626 02:03:09.125851 24207 raft_consensus.cc:1215] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.30->[1.31-1.31] Dedup: 1.31->[]
I20250626 02:03:09.135463 24283 raft_consensus.cc:1215] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20250626 02:03:09.177760 24507 raft_consensus.cc:991] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower d150470bce35446b9b3f41ec78cb5064 to start an election
I20250626 02:03:09.177920 24507 raft_consensus.cc:1079] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Signalling peer d150470bce35446b9b3f41ec78cb5064 to start an election
I20250626 02:03:09.178282 23892 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
from {username='slave'} at 127.0.0.1:37532
I20250626 02:03:09.178421 23892 raft_consensus.cc:491] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:09.178488 23892 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:09.179451 23892 raft_consensus.cc:513] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:09.179837 23892 leader_election.cc:290] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 3 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:09.180070 23739 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:09.180070 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:09.180298 23817 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:09.181337 23817 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 3.
I20250626 02:03:09.180280 23739 raft_consensus.cc:3053] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:03:09.181519 23739 raft_consensus.cc:738] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:09.181624 23858 leader_election.cc:304] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:09.181671 23739 consensus_queue.cc:260] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:09.181814 24515 raft_consensus.cc:2802] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:03:09.181876 23739 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:09.181896 24515 raft_consensus.cc:695] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Becoming Leader. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:09.182027 24515 consensus_queue.cc:237] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:09.182657 23739 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 3.
I20250626 02:03:09.183578 23635 catalog_manager.cc:5582] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: term changed from 2 to 3, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to d150470bce35446b9b3f41ec78cb5064 (127.19.220.131). New cstate: current_term: 3 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:09.240262 24132 raft_consensus.cc:3058] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Advancing to term 4
I20250626 02:03:09.241437 24132 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 4 LEARNER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 02:03:09.241830 24514 consensus_queue.cc:1037] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:09.243495 24514 raft_consensus.cc:1062] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80: attempting to promote NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd to VOTER
I20250626 02:03:09.243870 24514 consensus_queue.cc:237] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:09.245210 24132 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 4 LEARNER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250626 02:03:09.245251 23891 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250626 02:03:09.245576 24514 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.245693 24525 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.245752 23739 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250626 02:03:09.246555 24524 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.247925 24525 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 29 to 32, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.248324 23891 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.248687 23739 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.250221 23635 catalog_manager.cc:5582] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 29 to 32, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.250452 24132 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.253453 23817 consensus_queue.cc:237] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:09.254390 24132 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250626 02:03:09.254446 23739 raft_consensus.cc:1273] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250626 02:03:09.254702 24524 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:09.254909 24514 consensus_queue.cc:1035] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:09.256234 24524 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.256773 24132 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.256978 23739 raft_consensus.cc:2953] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.258014 23625 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e38ec4282a4b40e8bb9d07ef486659d6 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:09.259217 23634 catalog_manager.cc:5582] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd reported cstate change: config changed from index 32 to 33, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New cstate: current_term: 4 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.261595 23883 tablet_service.cc:1515] Processing DeleteTablet for tablet e38ec4282a4b40e8bb9d07ef486659d6 with delete_type TABLET_DATA_TOMBSTONED (TS d150470bce35446b9b3f41ec78cb5064 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:09.261982 24528 tablet_replica.cc:331] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:09.262132 24528 raft_consensus.cc:2241] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:09.262494 24528 raft_consensus.cc:2270] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:09.263206 24528 ts_tablet_manager.cc:1905] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:09.265098 24528 ts_tablet_manager.cc:1918] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.32
I20250626 02:03:09.265177 24528 log.cc:1199] T e38ec4282a4b40e8bb9d07ef486659d6 P d150470bce35446b9b3f41ec78cb5064: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/e38ec4282a4b40e8bb9d07ef486659d6
I20250626 02:03:09.265658 23623 catalog_manager.cc:4928] TS d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133): tablet e38ec4282a4b40e8bb9d07ef486659d6 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:09.445433 24525 raft_consensus.cc:991] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:09.445571 24525 raft_consensus.cc:1079] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:09.445967 23739 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40588
I20250626 02:03:09.446172 23739 raft_consensus.cc:491] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:09.446244 23739 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:09.447309 23739 raft_consensus.cc:513] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:09.447719 23739 leader_election.cc:290] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 3 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:09.448032 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:09.448181 23817 raft_consensus.cc:3053] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:03:09.448236 23817 raft_consensus.cc:738] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:09.448369 23817 consensus_queue.cc:260] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:09.448447 23891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:09.448526 23817 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:09.448540 23891 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:09.449240 23817 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 3.
I20250626 02:03:09.449247 23891 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 3.
I20250626 02:03:09.449612 23705 leader_election.cc:304] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:09.449836 24507 raft_consensus.cc:2802] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:03:09.449928 24507 raft_consensus.cc:695] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:09.450050 24507 consensus_queue.cc:237] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:09.451706 23635 catalog_manager.cc:5582] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 2 to 3, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 3 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:09.506631 24208 raft_consensus.cc:1215] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20250626 02:03:09.573540 24481 raft_consensus.cc:1062] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: attempting to promote NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd to VOTER
I20250626 02:03:09.573963 24481 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 32, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:09.575255 23739 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:09.575356 23817 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:09.575275 24132 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:09.575759 24533 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:09.575968 24532 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:09.576210 24481 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:09.577991 24532 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 32 to 34, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.578348 23817 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 32 to 34, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.578526 23739 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 32 to 34, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.580446 24132 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 32 to 34, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.580641 23635 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: config changed from index 32 to 34, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c5ee348337b146838a15e2d7161f5fb8" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:09.581542 24208 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:09.581944 24515 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.584164 24207 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:09.585456 23817 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:09.585650 23740 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20250626 02:03:09.585762 24502 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.586503 24533 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:09.587981 24132 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20250626 02:03:09.588384 24530 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:09.588622 24481 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.589030 23739 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.590240 23684 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 363d650f886244d38b78e6af7bb5f6ce with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:09.590938 23639 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: config changed from index 34 to 35, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New cstate: current_term: 2 leader_uuid: "c5ee348337b146838a15e2d7161f5fb8" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:09.590945 24132 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:09.594014 23807 tablet_service.cc:1515] Processing DeleteTablet for tablet 363d650f886244d38b78e6af7bb5f6ce with delete_type TABLET_DATA_TOMBSTONED (TS 6c2126b790c0477595d5b94c658b8a80 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:51988
I20250626 02:03:09.594471 24541 tablet_replica.cc:331] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:09.594650 24541 raft_consensus.cc:2241] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:09.594839 24541 raft_consensus.cc:2270] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:09.595490 24541 ts_tablet_manager.cc:1905] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:09.597506 24541 ts_tablet_manager.cc:1918] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.34
I20250626 02:03:09.597586 24541 log.cc:1199] T 363d650f886244d38b78e6af7bb5f6ce P 6c2126b790c0477595d5b94c658b8a80: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/363d650f886244d38b78e6af7bb5f6ce
I20250626 02:03:09.598028 23625 catalog_manager.cc:4928] TS 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183): tablet 363d650f886244d38b78e6af7bb5f6ce (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:09.742553 24515 raft_consensus.cc:1062] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: attempting to promote NON_VOTER c5ee348337b146838a15e2d7161f5fb8 to VOTER
I20250626 02:03:09.743053 24515 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:09.744467 23739 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250626 02:03:09.744863 24283 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250626 02:03:09.745031 24207 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250626 02:03:09.745004 24542 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.745380 24542 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.745613 24502 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.747599 24502 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:09.748229 24207 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:09.748371 24283 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:09.749773 23739 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:09.750262 23634 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: config changed from index 31 to 32, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:09.753917 23891 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:09.754935 24283 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250626 02:03:09.755033 24207 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250626 02:03:09.755349 24544 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:09.755663 24544 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:09.757140 24515 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:09.758417 23623 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 34ac2721a9bb44efa815122997c53fd7 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:09.758739 24283 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:09.759307 23635 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 32 to 33, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:09.760413 24207 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:09.762563 23727 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ac2721a9bb44efa815122997c53fd7 with delete_type TABLET_DATA_TOMBSTONED (TS 3a8897880da64721ad84d66ff1fcf3a4 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:40586
I20250626 02:03:09.762866 24548 tablet_replica.cc:331] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:09.763025 24548 raft_consensus.cc:2241] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:09.763366 24548 raft_consensus.cc:2270] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:09.764556 24548 ts_tablet_manager.cc:1905] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:09.766433 24548 ts_tablet_manager.cc:1918] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.32
I20250626 02:03:09.766511 24548 log.cc:1199] T 34ac2721a9bb44efa815122997c53fd7 P 3a8897880da64721ad84d66ff1fcf3a4: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/34ac2721a9bb44efa815122997c53fd7
I20250626 02:03:09.766983 23626 catalog_manager.cc:4928] TS 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725): tablet 34ac2721a9bb44efa815122997c53fd7 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:09.778621 23739 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 02:03:09.779029 24544 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:09.781832 23817 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 02:03:09.782212 24283 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Advancing to term 3
I20250626 02:03:09.782301 24542 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:09.783275 24283 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 3 LEARNER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 02:03:09.783695 24550 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:09.785687 24515 raft_consensus.cc:1062] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: attempting to promote NON_VOTER 46dfe87116554377a3d4bf78c9f9699b to VOTER
I20250626 02:03:09.786051 24515 consensus_queue.cc:237] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:09.787074 23817 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250626 02:03:09.787407 24502 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:09.787632 23739 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250626 02:03:09.787637 24283 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 3 LEARNER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250626 02:03:09.788002 24550 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:09.788269 24550 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:09.789536 24542 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:09.789896 23740 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:09.789927 24283 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:09.791986 23817 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:09.792084 23635 catalog_manager.cc:5582] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:09.796325 23892 consensus_queue.cc:237] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:09.797308 24283 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250626 02:03:09.797771 23740 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250626 02:03:09.797786 24550 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.798064 24515 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:09.799345 24550 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:09.799746 23740 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:09.799896 24283 raft_consensus.cc:2953] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:09.800956 23623 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 13501c41bba94a7bbfb48a2715cd2d40 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:09.802018 23639 catalog_manager.cc:5582] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 31 to 32, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New cstate: current_term: 3 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:09.804301 23807 tablet_service.cc:1515] Processing DeleteTablet for tablet 13501c41bba94a7bbfb48a2715cd2d40 with delete_type TABLET_DATA_TOMBSTONED (TS 6c2126b790c0477595d5b94c658b8a80 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:51988
I20250626 02:03:09.804508 24541 tablet_replica.cc:331] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:09.804652 24541 raft_consensus.cc:2241] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:09.804807 24541 raft_consensus.cc:2270] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:09.805361 24541 ts_tablet_manager.cc:1905] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:09.807276 24541 ts_tablet_manager.cc:1918] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20250626 02:03:09.807358 24541 log.cc:1199] T 13501c41bba94a7bbfb48a2715cd2d40 P 6c2126b790c0477595d5b94c658b8a80: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/13501c41bba94a7bbfb48a2715cd2d40
I20250626 02:03:09.807837 23625 catalog_manager.cc:4928] TS 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183): tablet 13501c41bba94a7bbfb48a2715cd2d40 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:09.898900 24500 tablet_copy_client.cc:538] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:09.900664 24500 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap starting.
I20250626 02:03:09.925794 24500 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=126 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:09.926136 24500 tablet_bootstrap.cc:492] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap complete.
I20250626 02:03:09.926331 24500 ts_tablet_manager.cc:1397] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.006s
I20250626 02:03:09.926817 24500 raft_consensus.cc:357] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:09.926949 24500 raft_consensus.cc:738] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 46dfe87116554377a3d4bf78c9f9699b, State: Initialized, Role: LEARNER
I20250626 02:03:09.927057 24500 consensus_queue.cc:260] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:09.927644 24500 ts_tablet_manager.cc:1428] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20250626 02:03:09.928614 23752 tablet_copy_service.cc:342] P 3a8897880da64721ad84d66ff1fcf3a4: Request end of tablet copy session 46dfe87116554377a3d4bf78c9f9699b-70698cc805a448b28cef88e47e146427 received from {username='slave'} at 127.0.0.1:40636
I20250626 02:03:09.928704 23752 tablet_copy_service.cc:434] P 3a8897880da64721ad84d66ff1fcf3a4: ending tablet copy session 46dfe87116554377a3d4bf78c9f9699b-70698cc805a448b28cef88e47e146427 on tablet 70698cc805a448b28cef88e47e146427 with peer 46dfe87116554377a3d4bf78c9f9699b
I20250626 02:03:09.946722 24283 raft_consensus.cc:1215] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Deduplicated request from leader. Original: 1.29->[2.30-2.30] Dedup: 2.30->[]
I20250626 02:03:10.007920 23817 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250626 02:03:10.008356 24507 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:10.011211 23892 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250626 02:03:10.011927 24558 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:10.022379 24502 raft_consensus.cc:1062] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: attempting to promote NON_VOTER c5ee348337b146838a15e2d7161f5fb8 to VOTER
I20250626 02:03:10.022814 24502 consensus_queue.cc:237] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:10.024106 24207 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 LEARNER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:10.024302 23739 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:10.024302 24132 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 02:03:10.024509 24550 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:10.024740 24544 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:10.024868 24502 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:10.026531 24550 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.027001 24132 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.027240 23739 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.028767 24207 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.029297 23634 catalog_manager.cc:5582] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index 29 to 30, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.032464 23891 consensus_queue.cc:237] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:10.033555 24207 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 02:03:10.033869 24502 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:10.034359 24132 raft_consensus.cc:1273] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 02:03:10.034773 24550 consensus_queue.cc:1035] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:10.035189 24550 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.035598 24207 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.036439 23623 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e37650dfb570400fb02742cb1220c7e1 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:10.036693 24132 raft_consensus.cc:2953] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.037787 23635 catalog_manager.cc:5582] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: config changed from index 30 to 31, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:10.039933 23727 tablet_service.cc:1515] Processing DeleteTablet for tablet e37650dfb570400fb02742cb1220c7e1 with delete_type TABLET_DATA_TOMBSTONED (TS 3a8897880da64721ad84d66ff1fcf3a4 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:40586
I20250626 02:03:10.040166 24548 tablet_replica.cc:331] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:10.040308 24548 raft_consensus.cc:2241] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:10.040481 24548 raft_consensus.cc:2270] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:10.041051 24548 ts_tablet_manager.cc:1905] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:10.043237 24548 ts_tablet_manager.cc:1918] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20250626 02:03:10.043318 24548 log.cc:1199] T e37650dfb570400fb02742cb1220c7e1 P 3a8897880da64721ad84d66ff1fcf3a4: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/e37650dfb570400fb02742cb1220c7e1
I20250626 02:03:10.043838 23626 catalog_manager.cc:4928] TS 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725): tablet e37650dfb570400fb02742cb1220c7e1 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:10.534608 24565 raft_consensus.cc:1062] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: attempting to promote NON_VOTER 46dfe87116554377a3d4bf78c9f9699b to VOTER
I20250626 02:03:10.535049 24565 consensus_queue.cc:237] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:10.536288 24283 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:10.536496 23891 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:10.536566 23817 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:10.536813 24559 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:10.537204 24559 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:10.537508 24559 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:10.539209 24567 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:10.539669 24283 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:10.540192 23891 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:10.542397 23639 catalog_manager.cc:5582] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:10.542574 23817 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:10.547562 23739 consensus_queue.cc:237] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:10.548599 24283 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:10.548624 23817 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:10.548969 24565 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:10.549167 24568 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:10.550710 24568 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:10.551047 24283 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:10.552326 23626 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 70698cc805a448b28cef88e47e146427 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:10.553184 23634 catalog_manager.cc:5582] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index 31 to 32, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New cstate: current_term: 2 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:10.553750 23817 raft_consensus.cc:2953] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:10.555243 23883 tablet_service.cc:1515] Processing DeleteTablet for tablet 70698cc805a448b28cef88e47e146427 with delete_type TABLET_DATA_TOMBSTONED (TS d150470bce35446b9b3f41ec78cb5064 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:10.555956 24575 tablet_replica.cc:331] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:10.556093 24575 raft_consensus.cc:2241] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:10.556306 24575 raft_consensus.cc:2270] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:10.556782 24575 ts_tablet_manager.cc:1905] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:10.558455 24575 ts_tablet_manager.cc:1918] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20250626 02:03:10.558532 24575 log.cc:1199] T 70698cc805a448b28cef88e47e146427 P d150470bce35446b9b3f41ec78cb5064: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/70698cc805a448b28cef88e47e146427
I20250626 02:03:10.559000 23623 catalog_manager.cc:4928] TS d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133): tablet 70698cc805a448b28cef88e47e146427 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:10.892551 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:10.894332 23892 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34ac2721a9bb44efa815122997c53fd7"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
mode: GRACEFUL
new_leader_uuid: "46dfe87116554377a3d4bf78c9f9699b"
from {username='slave'} at 127.0.0.1:47872
I20250626 02:03:10.894519 23892 raft_consensus.cc:604] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Received request to transfer leadership to TS 46dfe87116554377a3d4bf78c9f9699b
I20250626 02:03:10.895303 23892 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
mode: GRACEFUL
new_leader_uuid: "c5ee348337b146838a15e2d7161f5fb8"
from {username='slave'} at 127.0.0.1:47872
I20250626 02:03:10.895411 23892 raft_consensus.cc:604] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 2 LEADER]: Received request to transfer leadership to TS c5ee348337b146838a15e2d7161f5fb8
I20250626 02:03:10.896167 23740 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "70698cc805a448b28cef88e47e146427"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:10.896265 23740 raft_consensus.cc:604] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Received request to transfer leadership to TS 6c2126b790c0477595d5b94c658b8a80
I20250626 02:03:10.896624 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 02:03:10.896781 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:10.956183 24567 raft_consensus.cc:991] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:10.956295 24567 raft_consensus.cc:1079] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:10.956732 23817 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "70698cc805a448b28cef88e47e146427"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:51998
I20250626 02:03:10.956866 23817 raft_consensus.cc:491] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:10.956915 23817 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:10.957719 23817 raft_consensus.cc:513] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:10.958041 23817 leader_election.cc:290] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 3 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101)
I20250626 02:03:10.958323 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:10.958422 24283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "46dfe87116554377a3d4bf78c9f9699b"
I20250626 02:03:10.958429 23740 raft_consensus.cc:3053] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:03:10.958550 23740 raft_consensus.cc:738] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:10.958570 24283 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:10.958714 23740 consensus_queue.cc:260] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:10.958873 23740 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:10.959384 24283 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 3.
I20250626 02:03:10.959661 23781 leader_election.cc:304] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46dfe87116554377a3d4bf78c9f9699b, 6c2126b790c0477595d5b94c658b8a80; no voters:
I20250626 02:03:10.959733 23740 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 3.
I20250626 02:03:10.959843 24525 raft_consensus.cc:2802] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:03:10.960038 24525 raft_consensus.cc:695] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 3 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:10.960197 24525 consensus_queue.cc:237] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:10.961549 23634 catalog_manager.cc:5582] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 2 to 3, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 3 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:10.966694 24550 raft_consensus.cc:991] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064: : Instructing follower c5ee348337b146838a15e2d7161f5fb8 to start an election
I20250626 02:03:10.966778 24550 raft_consensus.cc:1079] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 2 LEADER]: Signalling peer c5ee348337b146838a15e2d7161f5fb8 to start an election
I20250626 02:03:10.967142 24207 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162"
dest_uuid: "c5ee348337b146838a15e2d7161f5fb8"
from {username='slave'} at 127.0.0.1:53580
I20250626 02:03:10.967269 24207 raft_consensus.cc:491] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:10.967344 24207 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:10.968173 24207 raft_consensus.cc:513] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:10.968588 24207 leader_election.cc:290] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [CANDIDATE]: Term 3 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:10.968917 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162" candidate_uuid: "c5ee348337b146838a15e2d7161f5fb8" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:10.968946 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65bf2a25991b4d4a94efbc900569a162" candidate_uuid: "c5ee348337b146838a15e2d7161f5fb8" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:10.969033 23892 raft_consensus.cc:3053] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:03:10.969043 23817 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:10.969094 23892 raft_consensus.cc:738] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:10.969240 23892 consensus_queue.cc:260] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:10.969421 23892 raft_consensus.cc:3058] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:10.969872 23817 raft_consensus.cc:2466] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ee348337b146838a15e2d7161f5fb8 in term 3.
I20250626 02:03:10.970155 24173 leader_election.cc:304] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, c5ee348337b146838a15e2d7161f5fb8; no voters:
I20250626 02:03:10.970170 23892 raft_consensus.cc:2466] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ee348337b146838a15e2d7161f5fb8 in term 3.
I20250626 02:03:10.970336 24576 raft_consensus.cc:2802] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:03:10.970537 24576 raft_consensus.cc:695] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 3 LEADER]: Becoming Leader. State: Replica: c5ee348337b146838a15e2d7161f5fb8, State: Running, Role: LEADER
I20250626 02:03:10.970655 24576 consensus_queue.cc:237] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:10.972021 23634 catalog_manager.cc:5582] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: term changed from 2 to 3, leader changed from d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) to c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133). New cstate: current_term: 3 leader_uuid: "c5ee348337b146838a15e2d7161f5fb8" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:11.237654 24578 raft_consensus.cc:991] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: : Instructing follower 46dfe87116554377a3d4bf78c9f9699b to start an election
I20250626 02:03:11.237769 24578 raft_consensus.cc:1079] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Signalling peer 46dfe87116554377a3d4bf78c9f9699b to start an election
I20250626 02:03:11.238196 24283 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "34ac2721a9bb44efa815122997c53fd7"
dest_uuid: "46dfe87116554377a3d4bf78c9f9699b"
from {username='slave'} at 127.0.0.1:54572
I20250626 02:03:11.238320 24283 raft_consensus.cc:491] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:11.238386 24283 raft_consensus.cc:3058] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:11.239228 24283 raft_consensus.cc:513] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:11.239691 24283 leader_election.cc:290] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [CANDIDATE]: Term 2 election: Requested vote from peers d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133), c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947)
I20250626 02:03:11.239976 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ac2721a9bb44efa815122997c53fd7" candidate_uuid: "46dfe87116554377a3d4bf78c9f9699b" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:11.240087 23892 raft_consensus.cc:3053] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:11.240157 23892 raft_consensus.cc:738] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:11.240301 23892 consensus_queue.cc:260] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 33, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:11.240476 23892 raft_consensus.cc:3058] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:11.241526 23892 raft_consensus.cc:2466] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46dfe87116554377a3d4bf78c9f9699b in term 2.
I20250626 02:03:11.241786 24246 leader_election.cc:304] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46dfe87116554377a3d4bf78c9f9699b, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:11.241987 24555 raft_consensus.cc:2802] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:11.242143 24555 raft_consensus.cc:695] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Becoming Leader. State: Replica: 46dfe87116554377a3d4bf78c9f9699b, State: Running, Role: LEADER
I20250626 02:03:11.242280 24555 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:11.243623 24207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ac2721a9bb44efa815122997c53fd7" candidate_uuid: "46dfe87116554377a3d4bf78c9f9699b" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: true dest_uuid: "c5ee348337b146838a15e2d7161f5fb8"
I20250626 02:03:11.243732 24207 raft_consensus.cc:3058] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:11.243784 23634 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: term changed from 1 to 2, leader changed from d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) to 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134). New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:11.244608 24207 raft_consensus.cc:2466] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46dfe87116554377a3d4bf78c9f9699b in term 2.
I20250626 02:03:11.449287 23740 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250626 02:03:11.449719 24525 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:11.452406 24283 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 3 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250626 02:03:11.452827 24586 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:11.562731 24132 consensus_queue.cc:237] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:11.564289 23817 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:11.564558 23892 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:11.564661 24592 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:11.564955 24504 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:11.566015 23818 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250626 02:03:11.566350 24579 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:11.566792 24592 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:11.567160 23892 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:11.569684 23634 catalog_manager.cc:5582] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd reported cstate change: config changed from index 30 to 32, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New cstate: current_term: 2 leader_uuid: "145a19f8089541e3845de3dc9a3ac9dd" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 02:03:11.570089 24095 consensus_peers.cc:487] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd -> Peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101): Couldn't send request to peer 46dfe87116554377a3d4bf78c9f9699b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 894851ab3c1e4d79971597e62b99cfbf. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:11.570183 23817 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:11.572052 23892 raft_consensus.cc:1273] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250626 02:03:11.572472 24579 consensus_queue.cc:1035] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:11.576125 24283 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:11.577415 23818 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:11.577669 23740 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:11.577739 24555 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:11.578032 24605 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:11.580037 24583 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:11.580494 23818 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
W20250626 02:03:11.581846 24249 consensus_peers.cc:487] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4a1bceeb5a4e3faba6c6ee9f5ec534. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:11.582146 23740 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:11.583194 23818 consensus_queue.cc:237] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:11.584048 23633 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index 30 to 32, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:11.584643 23740 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
W20250626 02:03:11.584798 23782 consensus_peers.cc:487] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 -> Peer c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947): Couldn't send request to peer c5ee348337b146838a15e2d7161f5fb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa3a5817958042e8b2753a3df8e2ba4c. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:11.584878 23892 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 02:03:11.585116 24586 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:11.585299 24577 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:11.586992 24586 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:11.587736 23740 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:11.588709 23892 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:11.589138 23635 catalog_manager.cc:5582] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index -1 to 30, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New cstate: current_term: 2 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:11.590261 24207 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 2.35, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:11.592288 24132 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250626 02:03:11.592607 24612 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 02:03:11.593230 23740 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250626 02:03:11.593551 24612 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 02:03:11.594372 24579 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEADER]: Committing config change with OpId 2.36: config changed from index 35 to 36, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
W20250626 02:03:11.596194 24171 consensus_peers.cc:487] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 -> Peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101): Couldn't send request to peer 46dfe87116554377a3d4bf78c9f9699b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 363d650f886244d38b78e6af7bb5f6ce. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:11.596664 24132 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
I20250626 02:03:11.596673 23635 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: config changed from index 35 to 36, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New cstate: current_term: 2 leader_uuid: "c5ee348337b146838a15e2d7161f5fb8" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 02:03:11.597059 23688 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20250626 02:03:11.598085 23740 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, NON_VOTER 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } } }
W20250626 02:03:11.601768 23688 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 34ac2721a9bb44efa815122997c53fd7, TS d150470bce35446b9b3f41ec78cb5064 -> TS 145a19f8089541e3845de3dc9a3ac9dd move failed, destination replica disappeared from tablet's Raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
W20250626 02:03:11.601819 23688 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 34ac2721a9bb44efa815122997c53fd7, TS d150470bce35446b9b3f41ec78cb5064 -> TS 145a19f8089541e3845de3dc9a3ac9dd move failed, destination replica disappeared from tablet's Raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
W20250626 02:03:11.605119 23688 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet dbd6b437781b4876af28add914954628, TS 6c2126b790c0477595d5b94c658b8a80 -> TS c5ee348337b146838a15e2d7161f5fb8 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
W20250626 02:03:11.605168 23688 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet dbd6b437781b4876af28add914954628, TS 6c2126b790c0477595d5b94c658b8a80 -> TS c5ee348337b146838a15e2d7161f5fb8 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:11.819314 23892 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:11.819720 24555 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:11.822077 24207 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:11.822432 24555 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:12.031240 24619 ts_tablet_manager.cc:927] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: Initiating tablet copy from peer c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947)
I20250626 02:03:12.031636 24619 tablet_copy_client.cc:323] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.133:43947
I20250626 02:03:12.032055 24218 tablet_copy_service.cc:140] P c5ee348337b146838a15e2d7161f5fb8: Received BeginTabletCopySession request for tablet 363d650f886244d38b78e6af7bb5f6ce from peer 46dfe87116554377a3d4bf78c9f9699b ({username='slave'} at 127.0.0.1:53592)
I20250626 02:03:12.032196 24218 tablet_copy_service.cc:161] P c5ee348337b146838a15e2d7161f5fb8: Beginning new tablet copy session on tablet 363d650f886244d38b78e6af7bb5f6ce from peer 46dfe87116554377a3d4bf78c9f9699b at {username='slave'} at 127.0.0.1:53592: session id = 46dfe87116554377a3d4bf78c9f9699b-363d650f886244d38b78e6af7bb5f6ce
I20250626 02:03:12.033214 24218 tablet_copy_source_session.cc:215] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:12.033972 24619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363d650f886244d38b78e6af7bb5f6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:12.035645 24621 ts_tablet_manager.cc:927] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: Initiating tablet copy from peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101)
I20250626 02:03:12.035938 24621 tablet_copy_client.cc:323] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.134:42101
I20250626 02:03:12.036371 24293 tablet_copy_service.cc:140] P 46dfe87116554377a3d4bf78c9f9699b: Received BeginTabletCopySession request for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 from peer 145a19f8089541e3845de3dc9a3ac9dd ({username='slave'} at 127.0.0.1:44528)
I20250626 02:03:12.036466 24293 tablet_copy_service.cc:161] P 46dfe87116554377a3d4bf78c9f9699b: Beginning new tablet copy session on tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 from peer 145a19f8089541e3845de3dc9a3ac9dd at {username='slave'} at 127.0.0.1:44528: session id = 145a19f8089541e3845de3dc9a3ac9dd-3a4a1bceeb5a4e3faba6c6ee9f5ec534
I20250626 02:03:12.036643 24619 tablet_copy_client.cc:806] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:12.036808 24619 tablet_copy_client.cc:670] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:12.037297 24293 tablet_copy_source_session.cc:215] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:12.037825 24620 tablet_copy_client.cc:1025] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:12.037979 24621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:12.039935 24621 tablet_copy_client.cc:806] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:12.040084 24621 tablet_copy_client.cc:670] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:12.041124 24622 tablet_copy_client.cc:1025] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:12.065908 24623 ts_tablet_manager.cc:927] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: Initiating tablet copy from peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409)
I20250626 02:03:12.066304 24623 tablet_copy_client.cc:323] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.132:33409
I20250626 02:03:12.066735 24142 tablet_copy_service.cc:140] P 145a19f8089541e3845de3dc9a3ac9dd: Received BeginTabletCopySession request for tablet 894851ab3c1e4d79971597e62b99cfbf from peer 46dfe87116554377a3d4bf78c9f9699b ({username='slave'} at 127.0.0.1:42494)
I20250626 02:03:12.066838 24142 tablet_copy_service.cc:161] P 145a19f8089541e3845de3dc9a3ac9dd: Beginning new tablet copy session on tablet 894851ab3c1e4d79971597e62b99cfbf from peer 46dfe87116554377a3d4bf78c9f9699b at {username='slave'} at 127.0.0.1:42494: session id = 46dfe87116554377a3d4bf78c9f9699b-894851ab3c1e4d79971597e62b99cfbf
I20250626 02:03:12.067759 24142 tablet_copy_source_session.cc:215] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:12.068622 24623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894851ab3c1e4d79971597e62b99cfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:12.070860 24623 tablet_copy_client.cc:806] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:12.071003 24623 tablet_copy_client.cc:670] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:12.072160 24624 tablet_copy_client.cc:1025] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:12.204789 24625 ts_tablet_manager.cc:927] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: Initiating tablet copy from peer 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:12.205107 24625 tablet_copy_client.cc:323] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:39183
I20250626 02:03:12.205502 23828 tablet_copy_service.cc:140] P 6c2126b790c0477595d5b94c658b8a80: Received BeginTabletCopySession request for tablet aa3a5817958042e8b2753a3df8e2ba4c from peer c5ee348337b146838a15e2d7161f5fb8 ({username='slave'} at 127.0.0.1:52060)
I20250626 02:03:12.205595 23828 tablet_copy_service.cc:161] P 6c2126b790c0477595d5b94c658b8a80: Beginning new tablet copy session on tablet aa3a5817958042e8b2753a3df8e2ba4c from peer c5ee348337b146838a15e2d7161f5fb8 at {username='slave'} at 127.0.0.1:52060: session id = c5ee348337b146838a15e2d7161f5fb8-aa3a5817958042e8b2753a3df8e2ba4c
I20250626 02:03:12.206418 23828 tablet_copy_source_session.cc:215] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:12.207288 24625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3a5817958042e8b2753a3df8e2ba4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:12.209851 24625 tablet_copy_client.cc:806] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:12.210067 24625 tablet_copy_client.cc:670] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:12.211056 24626 tablet_copy_client.cc:1025] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:12.897019 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:12.898911 23817 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "70698cc805a448b28cef88e47e146427"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:12.899058 23817 raft_consensus.cc:604] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 3 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:12.899446 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:12.899571 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:13.299342 24631 raft_consensus.cc:991] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:13.299499 24631 raft_consensus.cc:1079] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 3 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:13.299901 23740 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "70698cc805a448b28cef88e47e146427"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40588
I20250626 02:03:13.300066 23740 raft_consensus.cc:491] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:13.300168 23740 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:13.301312 23740 raft_consensus.cc:513] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:13.301731 23740 leader_election.cc:290] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 4 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101)
I20250626 02:03:13.302011 24283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "46dfe87116554377a3d4bf78c9f9699b"
I20250626 02:03:13.302011 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:13.302148 23817 raft_consensus.cc:3053] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 02:03:13.302148 24283 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:13.302215 23817 raft_consensus.cc:738] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:13.302361 23817 consensus_queue.cc:260] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:13.302520 23817 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:13.303161 24283 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 4.
I20250626 02:03:13.303429 23817 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 4.
I20250626 02:03:13.303442 23705 leader_election.cc:304] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 46dfe87116554377a3d4bf78c9f9699b; no voters:
I20250626 02:03:13.303764 24567 raft_consensus.cc:2802] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 02:03:13.303984 24567 raft_consensus.cc:695] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:13.304147 24567 consensus_queue.cc:237] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:13.305670 23639 catalog_manager.cc:5582] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 3 to 4, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 4 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:13.685011 23818 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250626 02:03:13.685385 24567 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:13.687701 24283 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 4 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250626 02:03:13.688016 24567 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:14.038120 24619 tablet_copy_client.cc:538] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:14.040982 24619 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap starting.
I20250626 02:03:14.041388 24621 tablet_copy_client.cc:538] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:14.042963 24621 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap starting.
I20250626 02:03:14.069535 24619 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:14.069954 24619 tablet_bootstrap.cc:492] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap complete.
I20250626 02:03:14.070169 24619 ts_tablet_manager.cc:1397] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.004s
I20250626 02:03:14.070741 24619 raft_consensus.cc:357] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:14.070917 24619 raft_consensus.cc:738] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 46dfe87116554377a3d4bf78c9f9699b, State: Initialized, Role: LEARNER
I20250626 02:03:14.071061 24619 consensus_queue.cc:260] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 2.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:14.071492 24619 ts_tablet_manager.cc:1428] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:14.071923 24218 tablet_copy_service.cc:342] P c5ee348337b146838a15e2d7161f5fb8: Request end of tablet copy session 46dfe87116554377a3d4bf78c9f9699b-363d650f886244d38b78e6af7bb5f6ce received from {username='slave'} at 127.0.0.1:53592
I20250626 02:03:14.072008 24218 tablet_copy_service.cc:434] P c5ee348337b146838a15e2d7161f5fb8: ending tablet copy session 46dfe87116554377a3d4bf78c9f9699b-363d650f886244d38b78e6af7bb5f6ce on tablet 363d650f886244d38b78e6af7bb5f6ce with peer 46dfe87116554377a3d4bf78c9f9699b
I20250626 02:03:14.072407 24623 tablet_copy_client.cc:538] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:14.073927 24623 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap starting.
I20250626 02:03:14.075263 24621 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:14.075539 24621 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap complete.
I20250626 02:03:14.075718 24621 ts_tablet_manager.cc:1397] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.009s
I20250626 02:03:14.076212 24621 raft_consensus.cc:357] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:14.076366 24621 raft_consensus.cc:738] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 145a19f8089541e3845de3dc9a3ac9dd, State: Initialized, Role: LEARNER
I20250626 02:03:14.076505 24621 consensus_queue.cc:260] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:14.077350 24621 ts_tablet_manager.cc:1428] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250626 02:03:14.078276 24293 tablet_copy_service.cc:342] P 46dfe87116554377a3d4bf78c9f9699b: Request end of tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-3a4a1bceeb5a4e3faba6c6ee9f5ec534 received from {username='slave'} at 127.0.0.1:44528
I20250626 02:03:14.078358 24293 tablet_copy_service.cc:434] P 46dfe87116554377a3d4bf78c9f9699b: ending tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-3a4a1bceeb5a4e3faba6c6ee9f5ec534 on tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with peer 145a19f8089541e3845de3dc9a3ac9dd
I20250626 02:03:14.081622 24283 raft_consensus.cc:1215] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.35->[2.36-2.36] Dedup: 2.36->[]
I20250626 02:03:14.099175 24623 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:14.099498 24623 tablet_bootstrap.cc:492] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: Bootstrap complete.
I20250626 02:03:14.099745 24623 ts_tablet_manager.cc:1397] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.005s
I20250626 02:03:14.100265 24623 raft_consensus.cc:357] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:14.100397 24623 raft_consensus.cc:738] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 46dfe87116554377a3d4bf78c9f9699b, State: Initialized, Role: LEARNER
I20250626 02:03:14.100490 24623 consensus_queue.cc:260] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: NON_VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: true } }
I20250626 02:03:14.100831 24623 ts_tablet_manager.cc:1428] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250626 02:03:14.101258 24142 tablet_copy_service.cc:342] P 145a19f8089541e3845de3dc9a3ac9dd: Request end of tablet copy session 46dfe87116554377a3d4bf78c9f9699b-894851ab3c1e4d79971597e62b99cfbf received from {username='slave'} at 127.0.0.1:42494
I20250626 02:03:14.101341 24142 tablet_copy_service.cc:434] P 145a19f8089541e3845de3dc9a3ac9dd: ending tablet copy session 46dfe87116554377a3d4bf78c9f9699b-894851ab3c1e4d79971597e62b99cfbf on tablet 894851ab3c1e4d79971597e62b99cfbf with peer 46dfe87116554377a3d4bf78c9f9699b
I20250626 02:03:14.151615 24132 raft_consensus.cc:1215] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20250626 02:03:14.158660 24283 raft_consensus.cc:1215] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20250626 02:03:14.211313 24625 tablet_copy_client.cc:538] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:14.213013 24625 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: Bootstrap starting.
I20250626 02:03:14.234790 24625 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:14.235136 24625 tablet_bootstrap.cc:492] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: Bootstrap complete.
I20250626 02:03:14.235348 24625 ts_tablet_manager.cc:1397] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.001s
I20250626 02:03:14.235793 24625 raft_consensus.cc:357] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:14.235899 24625 raft_consensus.cc:738] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c5ee348337b146838a15e2d7161f5fb8, State: Initialized, Role: LEARNER
I20250626 02:03:14.235994 24625 consensus_queue.cc:260] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:14.236335 24625 ts_tablet_manager.cc:1428] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20250626 02:03:14.236812 23828 tablet_copy_service.cc:342] P 6c2126b790c0477595d5b94c658b8a80: Request end of tablet copy session c5ee348337b146838a15e2d7161f5fb8-aa3a5817958042e8b2753a3df8e2ba4c received from {username='slave'} at 127.0.0.1:52060
I20250626 02:03:14.236891 23828 tablet_copy_service.cc:434] P 6c2126b790c0477595d5b94c658b8a80: ending tablet copy session c5ee348337b146838a15e2d7161f5fb8-aa3a5817958042e8b2753a3df8e2ba4c on tablet aa3a5817958042e8b2753a3df8e2ba4c with peer c5ee348337b146838a15e2d7161f5fb8
I20250626 02:03:14.254197 24208 raft_consensus.cc:1215] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20250626 02:03:14.530711 24656 raft_consensus.cc:1062] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: attempting to promote NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd to VOTER
I20250626 02:03:14.531191 24656 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:14.532634 24132 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 02:03:14.533094 23817 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 02:03:14.533457 24655 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:14.533710 23740 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 02:03:14.533988 24656 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:14.534300 24656 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:14.536577 24638 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.537021 23740 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.538861 23817 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.538947 24132 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.539682 23639 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: config changed from index 32 to 33, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.545811 24283 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:14.546769 23817 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:14.547063 24132 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:14.547550 24661 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:14.547729 24655 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:14.549136 24655 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.549505 24132 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.549863 23818 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.551910 23635 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd reported cstate change: config changed from index 33 to 34, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:14.552090 23682 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:14.554128 23727 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with delete_type TABLET_DATA_TOMBSTONED (TS 3a8897880da64721ad84d66ff1fcf3a4 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:40586
I20250626 02:03:14.554602 24666 tablet_replica.cc:331] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:14.554744 24666 raft_consensus.cc:2241] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:14.554947 24666 raft_consensus.cc:2270] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:14.555636 24666 ts_tablet_manager.cc:1905] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:14.557744 24666 ts_tablet_manager.cc:1918] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20250626 02:03:14.557809 24666 log.cc:1199] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 3a8897880da64721ad84d66ff1fcf3a4: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/3a4a1bceeb5a4e3faba6c6ee9f5ec534
I20250626 02:03:14.558280 23626 catalog_manager.cc:4928] TS 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725): tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:14.694937 24668 raft_consensus.cc:1062] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: attempting to promote NON_VOTER 46dfe87116554377a3d4bf78c9f9699b to VOTER
I20250626 02:03:14.695391 24668 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 32, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:14.696688 24283 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250626 02:03:14.696885 24131 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250626 02:03:14.697062 24636 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 02:03:14.697286 23740 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250626 02:03:14.697645 24636 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 02:03:14.697974 24636 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 02:03:14.699684 24636 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEADER]: Committing config change with OpId 2.37: config changed from index 36 to 37, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.700147 24131 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.700417 23740 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.701697 24283 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.702373 23634 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: config changed from index 36 to 37, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c5ee348337b146838a15e2d7161f5fb8" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:14.705284 24207 consensus_queue.cc:237] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:14.706264 24283 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
I20250626 02:03:14.706338 24131 raft_consensus.cc:1273] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer c5ee348337b146838a15e2d7161f5fb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
I20250626 02:03:14.706660 24667 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20250626 02:03:14.706828 24668 consensus_queue.cc:1035] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20250626 02:03:14.708109 24636 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEADER]: Committing config change with OpId 2.38: config changed from index 37 to 38, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.708637 24131 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index 37 to 38, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.709627 23684 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 363d650f886244d38b78e6af7bb5f6ce with cas_config_opid_index 37: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:14.709610 24283 raft_consensus.cc:2953] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index 37 to 38, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.711356 23635 catalog_manager.cc:5582] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd reported cstate change: config changed from index 37 to 38, VOTER 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) evicted. New cstate: current_term: 2 leader_uuid: "c5ee348337b146838a15e2d7161f5fb8" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.713575 23727 tablet_service.cc:1515] Processing DeleteTablet for tablet 363d650f886244d38b78e6af7bb5f6ce with delete_type TABLET_DATA_TOMBSTONED (TS 3a8897880da64721ad84d66ff1fcf3a4 not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:40586
I20250626 02:03:14.714131 24666 tablet_replica.cc:331] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:14.714278 24666 raft_consensus.cc:2241] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:14.714468 24666 raft_consensus.cc:2270] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:14.715269 24666 ts_tablet_manager.cc:1905] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:14.717273 24666 ts_tablet_manager.cc:1918] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.37
I20250626 02:03:14.717358 24666 log.cc:1199] T 363d650f886244d38b78e6af7bb5f6ce P 3a8897880da64721ad84d66ff1fcf3a4: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/363d650f886244d38b78e6af7bb5f6ce
I20250626 02:03:14.717851 23626 catalog_manager.cc:4928] TS 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725): tablet 363d650f886244d38b78e6af7bb5f6ce (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:14.758396 24676 raft_consensus.cc:1062] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: attempting to promote NON_VOTER 46dfe87116554377a3d4bf78c9f9699b to VOTER
I20250626 02:03:14.758891 24676 consensus_queue.cc:237] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:14.760346 24283 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEARNER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 02:03:14.760424 23818 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 02:03:14.760424 23892 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 02:03:14.760880 24675 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:14.761250 24678 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:14.761389 24676 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:14.763638 24678 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.764117 24283 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.764158 23818 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.766479 23892 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.766474 23639 catalog_manager.cc:5582] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: config changed from index 32 to 33, 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "145a19f8089541e3845de3dc9a3ac9dd" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.770231 24132 consensus_queue.cc:237] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:14.771726 24283 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:14.771975 23892 raft_consensus.cc:1273] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 145a19f8089541e3845de3dc9a3ac9dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250626 02:03:14.772051 24676 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:14.772360 24634 consensus_queue.cc:1035] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:14.773751 24676 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.774156 24283 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.775413 23686 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 894851ab3c1e4d79971597e62b99cfbf with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:14.775851 23635 catalog_manager.cc:5582] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd reported cstate change: config changed from index 33 to 34, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New cstate: current_term: 2 leader_uuid: "145a19f8089541e3845de3dc9a3ac9dd" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:14.776078 23892 raft_consensus.cc:2953] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } }
I20250626 02:03:14.778051 23807 tablet_service.cc:1515] Processing DeleteTablet for tablet 894851ab3c1e4d79971597e62b99cfbf with delete_type TABLET_DATA_TOMBSTONED (TS 6c2126b790c0477595d5b94c658b8a80 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:51988
I20250626 02:03:14.778507 24682 tablet_replica.cc:331] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:14.778685 24682 raft_consensus.cc:2241] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:14.778839 24682 raft_consensus.cc:2270] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:14.779577 24682 ts_tablet_manager.cc:1905] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:14.781461 24682 ts_tablet_manager.cc:1918] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20250626 02:03:14.781540 24682 log.cc:1199] T 894851ab3c1e4d79971597e62b99cfbf P 6c2126b790c0477595d5b94c658b8a80: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/894851ab3c1e4d79971597e62b99cfbf
I20250626 02:03:14.782007 23625 catalog_manager.cc:4928] TS 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183): tablet 894851ab3c1e4d79971597e62b99cfbf (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:14.857056 24631 raft_consensus.cc:1062] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: attempting to promote NON_VOTER c5ee348337b146838a15e2d7161f5fb8 to VOTER
I20250626 02:03:14.857446 24631 consensus_queue.cc:237] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:14.858799 23740 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:14.858807 23892 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:14.858834 24207 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEARNER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 02:03:14.859280 24685 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:14.859534 24686 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:14.859697 24683 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:14.861851 24686 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:14.862452 23892 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:14.863981 23740 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:14.864241 24207 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:14.864548 23634 catalog_manager.cc:5582] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index 30 to 31, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:14.868018 23818 consensus_queue.cc:237] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:14.869124 23740 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:14.869177 24207 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 02:03:14.869450 24685 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:14.869594 24631 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:14.870921 24683 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:14.871325 23740 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:14.871325 24207 raft_consensus.cc:2953] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:14.872556 23625 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet aa3a5817958042e8b2753a3df8e2ba4c with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:14.873149 23635 catalog_manager.cc:5582] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: config changed from index 31 to 32, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New cstate: current_term: 2 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:14.875195 23883 tablet_service.cc:1515] Processing DeleteTablet for tablet aa3a5817958042e8b2753a3df8e2ba4c with delete_type TABLET_DATA_TOMBSTONED (TS d150470bce35446b9b3f41ec78cb5064 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:14.875579 24690 tablet_replica.cc:331] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:14.875730 24690 raft_consensus.cc:2241] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:14.875872 24690 raft_consensus.cc:2270] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:14.876396 24690 ts_tablet_manager.cc:1905] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:14.878046 24690 ts_tablet_manager.cc:1918] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20250626 02:03:14.878109 24690 log.cc:1199] T aa3a5817958042e8b2753a3df8e2ba4c P d150470bce35446b9b3f41ec78cb5064: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/aa3a5817958042e8b2753a3df8e2ba4c
I20250626 02:03:14.878561 23623 catalog_manager.cc:4928] TS d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133): tablet aa3a5817958042e8b2753a3df8e2ba4c (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:14.899952 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:14.901723 23740 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:14.901885 23740 raft_consensus.cc:604] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Received request to transfer leadership to TS 6c2126b790c0477595d5b94c658b8a80
I20250626 02:03:14.902253 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:14.902392 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:14.967180 24653 raft_consensus.cc:991] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:14.967317 24653 raft_consensus.cc:1079] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:14.967765 23817 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:51998
I20250626 02:03:14.967914 23817 raft_consensus.cc:491] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:14.967988 23817 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:14.968852 23817 raft_consensus.cc:513] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:14.969182 23817 leader_election.cc:290] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 2 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:14.969478 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:14.969487 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:14.969583 23892 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:14.969643 23740 raft_consensus.cc:3053] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:14.969693 23740 raft_consensus.cc:738] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:14.969856 23740 consensus_queue.cc:260] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:14.970006 23740 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:14.970414 23892 raft_consensus.cc:2466] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 2.
I20250626 02:03:14.970705 23781 leader_election.cc:304] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:14.970855 24631 raft_consensus.cc:2802] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:14.970902 23740 raft_consensus.cc:2466] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 2.
I20250626 02:03:14.970924 24631 raft_consensus.cc:695] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:14.971078 24631 consensus_queue.cc:237] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:14.972527 23633 catalog_manager.cc:5582] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 1 to 2, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 2 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:15.457165 23740 raft_consensus.cc:1273] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 02:03:15.457580 24631 consensus_queue.cc:1035] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:15.460397 23892 raft_consensus.cc:1273] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 02:03:15.461346 24696 consensus_queue.cc:1035] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 02:03:15.877480 24283 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:15.878798 24132 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
W20250626 02:03:15.878892 24247 consensus_peers.cc:487] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b -> Peer c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947): Couldn't send request to peer c5ee348337b146838a15e2d7161f5fb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4a1bceeb5a4e3faba6c6ee9f5ec534. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:15.878899 23817 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20250626 02:03:15.879194 24655 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:15.879379 24701 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:15.881050 24702 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:15.881628 23817 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:15.881695 24132 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } } }
I20250626 02:03:15.883788 23639 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: config changed from index 34 to 35, NON_VOTER c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) added. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:15.884080 24283 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:15.885471 23892 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
W20250626 02:03:15.885664 24249 consensus_peers.cc:487] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b -> Peer 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132:33409): Couldn't send request to peer 145a19f8089541e3845de3dc9a3ac9dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34ac2721a9bb44efa815122997c53fd7. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:15.885747 24207 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20250626 02:03:15.885917 24655 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:15.886044 24702 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:15.887897 24702 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 33 to 35, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:15.888252 23892 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 33 to 35, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:15.888455 24207 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 33 to 35, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:15.890290 23635 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: config changed from index 33 to 35, NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) added. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } } }
I20250626 02:03:16.337741 24712 ts_tablet_manager.cc:927] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: Initiating tablet copy from peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101)
I20250626 02:03:16.338109 24712 tablet_copy_client.cc:323] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.134:42101
I20250626 02:03:16.338537 24293 tablet_copy_service.cc:140] P 46dfe87116554377a3d4bf78c9f9699b: Received BeginTabletCopySession request for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 from peer c5ee348337b146838a15e2d7161f5fb8 ({username='slave'} at 127.0.0.1:44542)
I20250626 02:03:16.338629 24293 tablet_copy_service.cc:161] P 46dfe87116554377a3d4bf78c9f9699b: Beginning new tablet copy session on tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 from peer c5ee348337b146838a15e2d7161f5fb8 at {username='slave'} at 127.0.0.1:44542: session id = c5ee348337b146838a15e2d7161f5fb8-3a4a1bceeb5a4e3faba6c6ee9f5ec534
I20250626 02:03:16.339387 24293 tablet_copy_source_session.cc:215] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:16.340176 24712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:16.342248 24712 tablet_copy_client.cc:806] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:16.342403 24712 tablet_copy_client.cc:670] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:16.343353 24713 tablet_copy_client.cc:1025] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:16.425235 24714 ts_tablet_manager.cc:927] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: Initiating tablet copy from peer 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101)
I20250626 02:03:16.425606 24714 tablet_copy_client.cc:323] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.134:42101
I20250626 02:03:16.426046 24293 tablet_copy_service.cc:140] P 46dfe87116554377a3d4bf78c9f9699b: Received BeginTabletCopySession request for tablet 34ac2721a9bb44efa815122997c53fd7 from peer 145a19f8089541e3845de3dc9a3ac9dd ({username='slave'} at 127.0.0.1:44528)
I20250626 02:03:16.426155 24293 tablet_copy_service.cc:161] P 46dfe87116554377a3d4bf78c9f9699b: Beginning new tablet copy session on tablet 34ac2721a9bb44efa815122997c53fd7 from peer 145a19f8089541e3845de3dc9a3ac9dd at {username='slave'} at 127.0.0.1:44528: session id = 145a19f8089541e3845de3dc9a3ac9dd-34ac2721a9bb44efa815122997c53fd7
I20250626 02:03:16.427158 24293 tablet_copy_source_session.cc:215] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:16.427917 24714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac2721a9bb44efa815122997c53fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:16.430135 24714 tablet_copy_client.cc:806] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:16.430344 24714 tablet_copy_client.cc:670] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:16.431258 24715 tablet_copy_client.cc:1025] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Injecting latency into file download: 2000
I20250626 02:03:16.902673 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:16.904420 23817 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "d150470bce35446b9b3f41ec78cb5064"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:16.904562 23817 raft_consensus.cc:604] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Received request to transfer leadership to TS d150470bce35446b9b3f41ec78cb5064
I20250626 02:03:16.904861 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:16.904994 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:16.919288 24685 raft_consensus.cc:991] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower d150470bce35446b9b3f41ec78cb5064 to start an election
I20250626 02:03:16.919451 24685 raft_consensus.cc:1079] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Signalling peer d150470bce35446b9b3f41ec78cb5064 to start an election
I20250626 02:03:16.919849 23892 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
from {username='slave'} at 127.0.0.1:37546
I20250626 02:03:16.919981 23892 raft_consensus.cc:491] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:16.920047 23892 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:16.920996 23892 raft_consensus.cc:513] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:16.921417 23892 leader_election.cc:290] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 3 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183)
I20250626 02:03:16.921723 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:16.921723 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "093974e120e643bc93fcb64da8f31ee1" candidate_uuid: "d150470bce35446b9b3f41ec78cb5064" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:16.921850 23740 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:16.921888 23817 raft_consensus.cc:3053] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:03:16.921939 23817 raft_consensus.cc:738] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:16.922076 23817 consensus_queue.cc:260] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:16.922214 23817 raft_consensus.cc:3058] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:16.922719 23740 raft_consensus.cc:2466] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 3.
I20250626 02:03:16.922981 23817 raft_consensus.cc:2466] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d150470bce35446b9b3f41ec78cb5064 in term 3.
I20250626 02:03:16.922992 23859 leader_election.cc:304] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:16.923163 24550 raft_consensus.cc:2802] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:03:16.923323 24550 raft_consensus.cc:695] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Becoming Leader. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:16.923483 24550 consensus_queue.cc:237] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:16.925004 23633 catalog_manager.cc:5582] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 reported cstate change: term changed from 2 to 3, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to d150470bce35446b9b3f41ec78cb5064 (127.19.220.131). New cstate: current_term: 3 leader_uuid: "d150470bce35446b9b3f41ec78cb5064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:17.432680 23817 raft_consensus.cc:1273] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 02:03:17.433089 24550 consensus_queue.cc:1035] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:17.435812 23740 raft_consensus.cc:1273] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Refusing update from remote peer d150470bce35446b9b3f41ec78cb5064: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 02:03:17.436223 24726 consensus_queue.cc:1035] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:18.343688 24712 tablet_copy_client.cc:538] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:18.345295 24712 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap starting.
I20250626 02:03:18.372951 24712 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:18.373286 24712 tablet_bootstrap.cc:492] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: Bootstrap complete.
I20250626 02:03:18.373515 24712 ts_tablet_manager.cc:1397] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: Time spent bootstrapping tablet: real 0.028s user 0.027s sys 0.000s
I20250626 02:03:18.374006 24712 raft_consensus.cc:357] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:18.374122 24712 raft_consensus.cc:738] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c5ee348337b146838a15e2d7161f5fb8, State: Initialized, Role: LEARNER
I20250626 02:03:18.374224 24712 consensus_queue.cc:260] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: NON_VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: true } }
I20250626 02:03:18.374503 24712 ts_tablet_manager.cc:1428] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:18.374913 24293 tablet_copy_service.cc:342] P 46dfe87116554377a3d4bf78c9f9699b: Request end of tablet copy session c5ee348337b146838a15e2d7161f5fb8-3a4a1bceeb5a4e3faba6c6ee9f5ec534 received from {username='slave'} at 127.0.0.1:44542
I20250626 02:03:18.375015 24293 tablet_copy_service.cc:434] P 46dfe87116554377a3d4bf78c9f9699b: ending tablet copy session c5ee348337b146838a15e2d7161f5fb8-3a4a1bceeb5a4e3faba6c6ee9f5ec534 on tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with peer c5ee348337b146838a15e2d7161f5fb8
I20250626 02:03:18.431540 24714 tablet_copy_client.cc:538] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:18.433416 24714 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap starting.
I20250626 02:03:18.458153 24714 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:18.458420 24714 tablet_bootstrap.cc:492] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: Bootstrap complete.
I20250626 02:03:18.458626 24714 ts_tablet_manager.cc:1397] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.003s
I20250626 02:03:18.459120 24714 raft_consensus.cc:357] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:18.459285 24714 raft_consensus.cc:738] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 145a19f8089541e3845de3dc9a3ac9dd, State: Initialized, Role: LEARNER
I20250626 02:03:18.459400 24714 consensus_queue.cc:260] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: true } }
I20250626 02:03:18.459690 24714 ts_tablet_manager.cc:1428] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250626 02:03:18.460271 24293 tablet_copy_service.cc:342] P 46dfe87116554377a3d4bf78c9f9699b: Request end of tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-34ac2721a9bb44efa815122997c53fd7 received from {username='slave'} at 127.0.0.1:44528
I20250626 02:03:18.460372 24293 tablet_copy_service.cc:434] P 46dfe87116554377a3d4bf78c9f9699b: ending tablet copy session 145a19f8089541e3845de3dc9a3ac9dd-34ac2721a9bb44efa815122997c53fd7 on tablet 34ac2721a9bb44efa815122997c53fd7 with peer 145a19f8089541e3845de3dc9a3ac9dd
I20250626 02:03:18.497550 24132 raft_consensus.cc:1215] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.34->[2.35-2.35] Dedup: 2.35->[]
I20250626 02:03:18.892542 24207 raft_consensus.cc:1215] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.34->[2.35-2.35] Dedup: 2.35->[]
I20250626 02:03:18.905272 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:18.907034 23892 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40"
dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:47872
I20250626 02:03:18.907186 23892 raft_consensus.cc:604] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:18.907545 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:18.907683 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:19.111613 24717 raft_consensus.cc:1062] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: attempting to promote NON_VOTER 145a19f8089541e3845de3dc9a3ac9dd to VOTER
I20250626 02:03:19.112027 24717 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 2.35, Last appended by leader: 33, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:19.113463 24207 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250626 02:03:19.113544 23892 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250626 02:03:19.113730 24132 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEARNER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250626 02:03:19.113821 24736 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 02:03:19.114909 24738 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 02:03:19.115096 24717 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 02:03:19.117257 24736 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.36: config changed from index 35 to 36, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:19.117591 24207 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:19.119112 23892 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:19.119434 24132 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:19.120121 23633 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 reported cstate change: config changed from index 35 to 36, 145a19f8089541e3845de3dc9a3ac9dd (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:19.125770 24282 consensus_queue.cc:237] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } }
I20250626 02:03:19.126816 24132 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250626 02:03:19.126914 24207 raft_consensus.cc:1273] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250626 02:03:19.127261 24738 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 02:03:19.127493 24736 consensus_queue.cc:1035] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 02:03:19.128700 24738 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:19.129693 24132 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:19.130409 23682 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 34ac2721a9bb44efa815122997c53fd7 with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:19.130764 24207 raft_consensus.cc:2953] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } }
I20250626 02:03:19.131508 23639 catalog_manager.cc:5582] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: config changed from index 36 to 37, VOTER d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) evicted. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:19.133889 23883 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ac2721a9bb44efa815122997c53fd7 with delete_type TABLET_DATA_TOMBSTONED (TS d150470bce35446b9b3f41ec78cb5064 not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:37526
I20250626 02:03:19.134269 24747 tablet_replica.cc:331] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:19.134392 24747 raft_consensus.cc:2241] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:19.134575 24747 raft_consensus.cc:2270] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:19.135149 24747 ts_tablet_manager.cc:1905] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:19.137416 24747 ts_tablet_manager.cc:1918] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.36
I20250626 02:03:19.137506 24747 log.cc:1199] T 34ac2721a9bb44efa815122997c53fd7 P d150470bce35446b9b3f41ec78cb5064: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/34ac2721a9bb44efa815122997c53fd7
I20250626 02:03:19.138011 23623 catalog_manager.cc:4928] TS d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133): tablet 34ac2721a9bb44efa815122997c53fd7 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:19.337563 24732 raft_consensus.cc:991] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:19.337687 24732 raft_consensus.cc:1079] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:19.338063 23740 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40596
I20250626 02:03:19.338193 23740 raft_consensus.cc:491] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:19.338266 23740 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:19.339373 23740 raft_consensus.cc:513] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:19.339825 23740 leader_election.cc:290] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 4 election: Requested vote from peers d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133), 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101)
I20250626 02:03:19.340020 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:19.340109 23892 raft_consensus.cc:3053] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 02:03:19.340178 23892 raft_consensus.cc:738] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d150470bce35446b9b3f41ec78cb5064, State: Running, Role: LEADER
I20250626 02:03:19.340314 23892 consensus_queue.cc:260] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:19.340417 24282 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13501c41bba94a7bbfb48a2715cd2d40" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "46dfe87116554377a3d4bf78c9f9699b"
I20250626 02:03:19.340462 23892 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:19.340502 24282 raft_consensus.cc:3058] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:19.341405 24282 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 4.
I20250626 02:03:19.341405 23892 raft_consensus.cc:2466] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 4.
I20250626 02:03:19.341737 23705 leader_election.cc:304] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, 46dfe87116554377a3d4bf78c9f9699b; no voters:
I20250626 02:03:19.341893 24653 raft_consensus.cc:2802] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 02:03:19.342105 24653 raft_consensus.cc:695] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:19.342247 24653 consensus_queue.cc:237] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:19.343703 23635 catalog_manager.cc:5582] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 3 to 4, leader changed from d150470bce35446b9b3f41ec78cb5064 (127.19.220.131) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 4 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:19.367319 24717 raft_consensus.cc:1062] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: attempting to promote NON_VOTER c5ee348337b146838a15e2d7161f5fb8 to VOTER
I20250626 02:03:19.367771 24717 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 2.35, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:19.368906 24132 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250626 02:03:19.369025 23817 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250626 02:03:19.369282 24207 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEARNER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250626 02:03:19.369356 24739 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 02:03:19.369557 24736 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 02:03:19.369678 24738 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 02:03:19.371790 24736 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.36: config changed from index 35 to 36, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:19.372110 23817 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:19.372346 24207 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:19.374382 23635 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: config changed from index 35 to 36, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:19.374953 24132 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } attrs { replace: true } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:19.377393 24282 consensus_queue.cc:237] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:19.378544 24207 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250626 02:03:19.378808 24132 raft_consensus.cc:1273] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Refusing update from remote peer 46dfe87116554377a3d4bf78c9f9699b: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250626 02:03:19.378877 24738 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 02:03:19.379201 24738 consensus_queue.cc:1035] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [LEADER]: Connected to new peer: Peer: permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 02:03:19.380458 24736 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:19.380861 24132 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:19.380844 24207 raft_consensus.cc:2953] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } } }
I20250626 02:03:19.381960 23682 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:19.382671 23635 catalog_manager.cc:5582] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b reported cstate change: config changed from index 36 to 37, VOTER 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) evicted. New cstate: current_term: 2 leader_uuid: "46dfe87116554377a3d4bf78c9f9699b" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "145a19f8089541e3845de3dc9a3ac9dd" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:03:19.384821 23807 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 with delete_type TABLET_DATA_TOMBSTONED (TS 6c2126b790c0477595d5b94c658b8a80 not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:51988
I20250626 02:03:19.385426 24755 tablet_replica.cc:331] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:19.385560 24755 raft_consensus.cc:2241] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:19.385773 24755 raft_consensus.cc:2270] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:19.386312 24755 ts_tablet_manager.cc:1905] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:19.388015 24755 ts_tablet_manager.cc:1918] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.36
I20250626 02:03:19.388085 24755 log.cc:1199] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 6c2126b790c0477595d5b94c658b8a80: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/3a4a1bceeb5a4e3faba6c6ee9f5ec534
I20250626 02:03:19.388571 23625 catalog_manager.cc:4928] TS 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183): tablet 3a4a1bceeb5a4e3faba6c6ee9f5ec534 (table test-workload [id=3cdc82a8101841d69b9a390998ae5d9f]) successfully deleted
I20250626 02:03:19.758839 23892 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250626 02:03:19.759254 24653 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:19.762082 24283 raft_consensus.cc:1273] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 4 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250626 02:03:19.762805 24749 consensus_queue.cc:1035] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:20.907951 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:20.909576 23740 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:20.909706 23740 raft_consensus.cc:604] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Received request to transfer leadership to TS 6c2126b790c0477595d5b94c658b8a80
I20250626 02:03:20.910055 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:20.910161 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:21.459254 24762 raft_consensus.cc:991] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:21.459358 24762 raft_consensus.cc:1079] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:21.459769 23817 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:51998
I20250626 02:03:21.459906 23817 raft_consensus.cc:491] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:21.459975 23817 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:21.460906 23817 raft_consensus.cc:513] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:21.461328 23817 leader_election.cc:290] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 4 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:21.461637 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:21.461637 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:21.461781 23740 raft_consensus.cc:3053] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 02:03:21.461781 23892 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:21.461859 23740 raft_consensus.cc:738] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:21.462018 23740 consensus_queue.cc:260] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:21.462177 23740 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:21.462836 23892 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 4.
I20250626 02:03:21.463122 23740 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 4.
I20250626 02:03:21.463130 23781 leader_election.cc:304] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:21.463321 24685 raft_consensus.cc:2802] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 02:03:21.463492 24685 raft_consensus.cc:695] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:21.463635 24685 consensus_queue.cc:237] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:21.465045 23634 catalog_manager.cc:5582] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 3 to 4, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 4 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:22.045637 23740 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250626 02:03:22.046021 24685 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:22.048627 23892 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250626 02:03:22.048998 24775 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 02:03:22.910547 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:22.912307 23817 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:22.912451 23817 raft_consensus.cc:604] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:22.912832 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:22.912958 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:23.389011 24775 raft_consensus.cc:991] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:23.389130 24775 raft_consensus.cc:1079] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:23.389564 23740 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40588
I20250626 02:03:23.389709 23740 raft_consensus.cc:491] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:23.389786 23740 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:23.391050 23740 raft_consensus.cc:513] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:23.391569 23740 leader_election.cc:290] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 3 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947)
I20250626 02:03:23.391902 24207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "c5ee348337b146838a15e2d7161f5fb8"
I20250626 02:03:23.391912 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:23.392017 24207 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:23.392050 23817 raft_consensus.cc:3053] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:03:23.392091 23817 raft_consensus.cc:738] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:23.392253 23817 consensus_queue.cc:260] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:23.392390 23817 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:03:23.393049 24207 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 3.
I20250626 02:03:23.393307 23706 leader_election.cc:304] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, c5ee348337b146838a15e2d7161f5fb8; no voters:
I20250626 02:03:23.393488 23817 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 3.
I20250626 02:03:23.393530 24762 raft_consensus.cc:2802] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:03:23.393685 24762 raft_consensus.cc:695] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:23.393821 24762 consensus_queue.cc:237] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:23.395210 23639 catalog_manager.cc:5582] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 2 to 3, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 3 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:23.947779 23817 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250626 02:03:23.948249 24762 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:23.950778 24207 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 3 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250626 02:03:23.951160 24762 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 02:03:24.913268 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:24.915062 23740 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:24.915218 23740 raft_consensus.cc:604] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Received request to transfer leadership to TS 6c2126b790c0477595d5b94c658b8a80
I20250626 02:03:24.915620 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:24.915768 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:25.412165 24793 raft_consensus.cc:991] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:25.412281 24793 raft_consensus.cc:1079] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:25.412679 23817 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:51998
I20250626 02:03:25.412817 23817 raft_consensus.cc:491] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:25.412890 23817 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:25.413753 23817 raft_consensus.cc:513] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:25.414155 23817 leader_election.cc:290] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 4 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947)
I20250626 02:03:25.414458 24207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "c5ee348337b146838a15e2d7161f5fb8"
I20250626 02:03:25.414481 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:25.414556 24207 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:25.414642 23740 raft_consensus.cc:3053] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 02:03:25.414710 23740 raft_consensus.cc:738] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:25.414881 23740 consensus_queue.cc:260] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:25.415036 23740 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Advancing to term 4
I20250626 02:03:25.415460 24207 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 4.
I20250626 02:03:25.415793 23782 leader_election.cc:304] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, c5ee348337b146838a15e2d7161f5fb8; no voters:
I20250626 02:03:25.415969 24775 raft_consensus.cc:2802] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 02:03:25.415957 23740 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 4.
I20250626 02:03:25.416229 24775 raft_consensus.cc:695] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:25.416600 24775 consensus_queue.cc:237] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:25.418056 23634 catalog_manager.cc:5582] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 3 to 4, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 4 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:25.922410 23740 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250626 02:03:25.922860 24775 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:25.925215 24207 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 4 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250626 02:03:25.925631 24775 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 02:03:26.916052 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:26.917861 23817 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:26.918012 23817 raft_consensus.cc:604] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:26.918349 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:26.918467 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:27.229614 24808 raft_consensus.cc:991] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:27.229749 24808 raft_consensus.cc:1079] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:27.230149 23740 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40588
I20250626 02:03:27.230273 23740 raft_consensus.cc:491] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:27.230340 23740 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:27.231221 23740 raft_consensus.cc:513] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:27.231580 23740 leader_election.cc:290] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 5 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:27.231854 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:27.231891 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:27.231993 23817 raft_consensus.cc:3053] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 02:03:27.232007 23892 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:27.232046 23817 raft_consensus.cc:738] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:27.232215 23817 consensus_queue.cc:260] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:27.232349 23817 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:27.232877 23892 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 5.
I20250626 02:03:27.233103 23817 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 5.
I20250626 02:03:27.233155 23705 leader_election.cc:304] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:27.233309 24793 raft_consensus.cc:2802] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 02:03:27.233441 24793 raft_consensus.cc:695] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:27.233565 24793 consensus_queue.cc:237] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:27.234902 23639 catalog_manager.cc:5582] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 4 to 5, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 5 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:27.634790 23892 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 5 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20250626 02:03:27.635179 24793 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:27.637542 23817 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20250626 02:03:27.637979 24810 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 02:03:28.918740 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:28.920558 23740 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "70698cc805a448b28cef88e47e146427"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:28.920709 23740 raft_consensus.cc:604] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 LEADER]: Received request to transfer leadership to TS 6c2126b790c0477595d5b94c658b8a80
I20250626 02:03:28.921085 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:28.921198 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:29.246955 24810 raft_consensus.cc:991] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:29.247062 24810 raft_consensus.cc:1079] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:29.247562 23818 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "70698cc805a448b28cef88e47e146427"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:51998
I20250626 02:03:29.247718 23818 raft_consensus.cc:491] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:29.247771 23818 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:29.248740 23818 raft_consensus.cc:513] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:29.249118 23818 leader_election.cc:290] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 5 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), 46dfe87116554377a3d4bf78c9f9699b (127.19.220.134:42101)
I20250626 02:03:29.249421 24283 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "46dfe87116554377a3d4bf78c9f9699b"
I20250626 02:03:29.249436 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70698cc805a448b28cef88e47e146427" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:29.249541 24283 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:29.249575 23740 raft_consensus.cc:3053] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 02:03:29.249629 23740 raft_consensus.cc:738] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:29.249780 23740 consensus_queue.cc:260] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:29.249922 23740 raft_consensus.cc:3058] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:29.250422 24283 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 5.
I20250626 02:03:29.250736 23781 leader_election.cc:304] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46dfe87116554377a3d4bf78c9f9699b, 6c2126b790c0477595d5b94c658b8a80; no voters:
I20250626 02:03:29.250799 23740 raft_consensus.cc:2466] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 5.
I20250626 02:03:29.250916 24808 raft_consensus.cc:2802] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 02:03:29.251070 24808 raft_consensus.cc:695] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 5 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:29.251211 24808 consensus_queue.cc:237] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } }
I20250626 02:03:29.252954 23634 catalog_manager.cc:5582] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 4 to 5, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 5 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:29.636974 23740 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20250626 02:03:29.637382 24826 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:29.639796 24283 raft_consensus.cc:1273] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 5 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20250626 02:03:29.640241 24826 consensus_queue.cc:1035] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dfe87116554377a3d4bf78c9f9699b" member_type: VOTER last_known_addr { host: "127.19.220.134" port: 42101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:30.921459 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:30.923048 23818 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
mode: GRACEFUL
new_leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:56812
I20250626 02:03:30.923180 23818 raft_consensus.cc:604] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Received request to transfer leadership to TS 3a8897880da64721ad84d66ff1fcf3a4
I20250626 02:03:30.923542 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:30.923641 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:31.115047 24837 raft_consensus.cc:991] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: : Instructing follower 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:31.115165 24837 raft_consensus.cc:1079] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Signalling peer 3a8897880da64721ad84d66ff1fcf3a4 to start an election
I20250626 02:03:31.115579 23740 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
from {username='slave'} at 127.0.0.1:40588
I20250626 02:03:31.115717 23740 raft_consensus.cc:491] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:31.115782 23740 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:31.116806 23740 raft_consensus.cc:513] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:31.117163 23740 leader_election.cc:290] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 5 election: Requested vote from peers 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130:39183), c5ee348337b146838a15e2d7161f5fb8 (127.19.220.133:43947)
I20250626 02:03:31.117514 24208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "c5ee348337b146838a15e2d7161f5fb8"
I20250626 02:03:31.117563 23818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa3a5817958042e8b2753a3df8e2ba4c" candidate_uuid: "3a8897880da64721ad84d66ff1fcf3a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
I20250626 02:03:31.117621 24208 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:31.117666 23818 raft_consensus.cc:3053] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 02:03:31.117724 23818 raft_consensus.cc:738] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:31.117861 23818 consensus_queue.cc:260] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:31.118001 23818 raft_consensus.cc:3058] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Advancing to term 5
I20250626 02:03:31.118496 24208 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 5.
I20250626 02:03:31.118723 23818 raft_consensus.cc:2466] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8897880da64721ad84d66ff1fcf3a4 in term 5.
I20250626 02:03:31.118789 23706 leader_election.cc:304] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8897880da64721ad84d66ff1fcf3a4, c5ee348337b146838a15e2d7161f5fb8; no voters:
I20250626 02:03:31.118932 24810 raft_consensus.cc:2802] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 02:03:31.119091 24810 raft_consensus.cc:695] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 LEADER]: Becoming Leader. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:31.119226 24810 consensus_queue.cc:237] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } }
I20250626 02:03:31.120594 23639 catalog_manager.cc:5582] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 reported cstate change: term changed from 4 to 5, leader changed from 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130) to 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129). New cstate: current_term: 5 leader_uuid: "3a8897880da64721ad84d66ff1fcf3a4" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:31.497028 23818 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20250626 02:03:31.497493 24810 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:31.500329 24208 raft_consensus.cc:1273] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 5 FOLLOWER]: Refusing update from remote peer 3a8897880da64721ad84d66ff1fcf3a4: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20250626 02:03:31.500763 24847 consensus_queue.cc:1035] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ee348337b146838a15e2d7161f5fb8" member_type: VOTER last_known_addr { host: "127.19.220.133" port: 43947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 02:03:32.924077 23693 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:32.925884 23740 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
mode: GRACEFUL
new_leader_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:32942
I20250626 02:03:32.926033 23740 raft_consensus.cc:604] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 LEADER]: Received request to transfer leadership to TS 6c2126b790c0477595d5b94c658b8a80
I20250626 02:03:32.926433 23693 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:32.926564 23693 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:33.354816 24847 raft_consensus.cc:991] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: : Instructing follower 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:33.354940 24847 raft_consensus.cc:1079] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 LEADER]: Signalling peer 6c2126b790c0477595d5b94c658b8a80 to start an election
I20250626 02:03:33.355353 23818 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dbd6b437781b4876af28add914954628"
dest_uuid: "6c2126b790c0477595d5b94c658b8a80"
from {username='slave'} at 127.0.0.1:51998
I20250626 02:03:33.355512 23818 raft_consensus.cc:491] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:33.355583 23818 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Advancing to term 6
I20250626 02:03:33.356477 23818 raft_consensus.cc:513] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:33.356848 23818 leader_election.cc:290] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 6 election: Requested vote from peers 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725), d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133)
I20250626 02:03:33.357122 23892 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 6 candidate_status { last_received { term: 5 index: 31 } } ignore_live_leader: true dest_uuid: "d150470bce35446b9b3f41ec78cb5064"
I20250626 02:03:33.357148 23740 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd6b437781b4876af28add914954628" candidate_uuid: "6c2126b790c0477595d5b94c658b8a80" candidate_term: 6 candidate_status { last_received { term: 5 index: 31 } } ignore_live_leader: true dest_uuid: "3a8897880da64721ad84d66ff1fcf3a4"
I20250626 02:03:33.357234 23892 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 5 FOLLOWER]: Advancing to term 6
I20250626 02:03:33.357280 23740 raft_consensus.cc:3053] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 02:03:33.357328 23740 raft_consensus.cc:738] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a8897880da64721ad84d66ff1fcf3a4, State: Running, Role: LEADER
I20250626 02:03:33.357488 23740 consensus_queue.cc:260] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:33.357645 23740 raft_consensus.cc:3058] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Advancing to term 6
I20250626 02:03:33.358132 23892 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 6.
I20250626 02:03:33.358398 23781 leader_election.cc:304] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2126b790c0477595d5b94c658b8a80, d150470bce35446b9b3f41ec78cb5064; no voters:
I20250626 02:03:33.358438 23740 raft_consensus.cc:2466] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2126b790c0477595d5b94c658b8a80 in term 6.
I20250626 02:03:33.358563 24837 raft_consensus.cc:2802] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 02:03:33.358728 24837 raft_consensus.cc:695] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 6 LEADER]: Becoming Leader. State: Replica: 6c2126b790c0477595d5b94c658b8a80, State: Running, Role: LEADER
I20250626 02:03:33.358889 24837 consensus_queue.cc:237] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } }
I20250626 02:03:33.360358 23639 catalog_manager.cc:5582] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 reported cstate change: term changed from 5 to 6, leader changed from 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129) to 6c2126b790c0477595d5b94c658b8a80 (127.19.220.130). New cstate: current_term: 6 leader_uuid: "6c2126b790c0477595d5b94c658b8a80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2126b790c0477595d5b94c658b8a80" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:33.853688 23892 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 6 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20250626 02:03:33.854151 24837 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d150470bce35446b9b3f41ec78cb5064" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 02:03:33.857054 23740 raft_consensus.cc:1273] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 6 FOLLOWER]: Refusing update from remote peer 6c2126b790c0477595d5b94c658b8a80: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20250626 02:03:33.857664 24862 consensus_queue.cc:1035] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8897880da64721ad84d66ff1fcf3a4" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:196: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250626 02:03:34.441219 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:03:34.445576 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:34.445791 20338 tablet_replica.cc:331] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:34.445924 20338 raft_consensus.cc:2241] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.446112 20338 raft_consensus.cc:2270] T 13501c41bba94a7bbfb48a2715cd2d40 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.446588 20338 tablet_replica.cc:331] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:34.446691 20338 raft_consensus.cc:2241] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.446787 20338 raft_consensus.cc:2270] T 70698cc805a448b28cef88e47e146427 P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.447083 20338 tablet_replica.cc:331] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:34.447173 20338 raft_consensus.cc:2241] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.447280 20338 raft_consensus.cc:2270] T e38ec4282a4b40e8bb9d07ef486659d6 P 3a8897880da64721ad84d66ff1fcf3a4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.447567 20338 tablet_replica.cc:331] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:34.447655 20338 raft_consensus.cc:2241] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.447795 20338 raft_consensus.cc:2270] T aa3a5817958042e8b2753a3df8e2ba4c P 3a8897880da64721ad84d66ff1fcf3a4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.448156 20338 tablet_replica.cc:331] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:34.448244 20338 raft_consensus.cc:2241] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.448326 20338 raft_consensus.cc:2270] T dbd6b437781b4876af28add914954628 P 3a8897880da64721ad84d66ff1fcf3a4 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.448745 20338 tablet_replica.cc:331] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4: stopping tablet replica
I20250626 02:03:34.448829 20338 raft_consensus.cc:2241] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.448911 20338 raft_consensus.cc:2270] T 093974e120e643bc93fcb64da8f31ee1 P 3a8897880da64721ad84d66ff1fcf3a4 [term 3 FOLLOWER]: Raft consensus is shut down!
W20250626 02:03:34.462096 23859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34725: connect: Connection refused (error 111)
W20250626 02:03:34.462873 23859 consensus_peers.cc:487] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 -> Peer 3a8897880da64721ad84d66ff1fcf3a4 (127.19.220.129:34725): Couldn't send request to peer 3a8897880da64721ad84d66ff1fcf3a4. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:34.464497 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:03:34.467473 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:03:34.472662 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:34.472848 20338 tablet_replica.cc:331] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:34.472957 20338 raft_consensus.cc:2241] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.473058 20338 raft_consensus.cc:2270] T 093974e120e643bc93fcb64da8f31ee1 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.473466 20338 tablet_replica.cc:331] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:34.473547 20338 raft_consensus.cc:2241] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.473641 20338 raft_consensus.cc:2270] T 65bf2a25991b4d4a94efbc900569a162 P 6c2126b790c0477595d5b94c658b8a80 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.473932 20338 tablet_replica.cc:331] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:34.474306 20338 raft_consensus.cc:2241] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 6 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.474472 20338 raft_consensus.cc:2270] T dbd6b437781b4876af28add914954628 P 6c2126b790c0477595d5b94c658b8a80 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.474924 20338 tablet_replica.cc:331] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:34.475016 20338 raft_consensus.cc:2241] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 4 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.475176 20338 raft_consensus.cc:2270] T e38ec4282a4b40e8bb9d07ef486659d6 P 6c2126b790c0477595d5b94c658b8a80 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.475495 20338 tablet_replica.cc:331] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:34.475561 20338 raft_consensus.cc:2241] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.475628 20338 raft_consensus.cc:2270] T aa3a5817958042e8b2753a3df8e2ba4c P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.475900 20338 tablet_replica.cc:331] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80: stopping tablet replica
I20250626 02:03:34.475960 20338 raft_consensus.cc:2241] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 5 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.476070 20338 raft_consensus.cc:2270] T 70698cc805a448b28cef88e47e146427 P 6c2126b790c0477595d5b94c658b8a80 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.479758 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:03:34.482746 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:03:34.488026 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:34.488242 20338 tablet_replica.cc:331] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:34.488369 20338 raft_consensus.cc:2241] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.488500 20338 raft_consensus.cc:2270] T 13501c41bba94a7bbfb48a2715cd2d40 P d150470bce35446b9b3f41ec78cb5064 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.489018 20338 tablet_replica.cc:331] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:34.489121 20338 raft_consensus.cc:2241] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.489212 20338 raft_consensus.cc:2270] T 65bf2a25991b4d4a94efbc900569a162 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.489538 20338 tablet_replica.cc:331] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:34.489614 20338 raft_consensus.cc:2241] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.489723 20338 raft_consensus.cc:2270] T dbd6b437781b4876af28add914954628 P d150470bce35446b9b3f41ec78cb5064 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.490120 20338 tablet_replica.cc:331] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:34.490193 20338 raft_consensus.cc:2241] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.490355 20338 raft_consensus.cc:2270] T e37650dfb570400fb02742cb1220c7e1 P d150470bce35446b9b3f41ec78cb5064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.490669 20338 tablet_replica.cc:331] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:34.490747 20338 raft_consensus.cc:2241] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 3 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.490883 20338 raft_consensus.cc:2270] T 093974e120e643bc93fcb64da8f31ee1 P d150470bce35446b9b3f41ec78cb5064 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.491173 20338 tablet_replica.cc:331] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064: stopping tablet replica
I20250626 02:03:34.491252 20338 raft_consensus.cc:2241] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.491398 20338 raft_consensus.cc:2270] T 894851ab3c1e4d79971597e62b99cfbf P d150470bce35446b9b3f41ec78cb5064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.505591 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:03:34.509047 20338 tablet_server.cc:178] TabletServer@127.19.220.132:0 shutting down...
I20250626 02:03:34.513715 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:34.513899 20338 tablet_replica.cc:331] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd: stopping tablet replica
I20250626 02:03:34.514061 20338 raft_consensus.cc:2241] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.514217 20338 raft_consensus.cc:2270] T 34ac2721a9bb44efa815122997c53fd7 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.514653 20338 tablet_replica.cc:331] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd: stopping tablet replica
I20250626 02:03:34.514746 20338 raft_consensus.cc:2241] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.514854 20338 raft_consensus.cc:2270] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.515142 20338 tablet_replica.cc:331] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd: stopping tablet replica
I20250626 02:03:34.515211 20338 raft_consensus.cc:2241] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.515305 20338 raft_consensus.cc:2270] T 363d650f886244d38b78e6af7bb5f6ce P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.515570 20338 tablet_replica.cc:331] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd: stopping tablet replica
I20250626 02:03:34.515642 20338 raft_consensus.cc:2241] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.515743 20338 raft_consensus.cc:2270] T e38ec4282a4b40e8bb9d07ef486659d6 P 145a19f8089541e3845de3dc9a3ac9dd [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.516013 20338 tablet_replica.cc:331] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd: stopping tablet replica
I20250626 02:03:34.516080 20338 raft_consensus.cc:2241] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.516278 20338 raft_consensus.cc:2270] T 894851ab3c1e4d79971597e62b99cfbf P 145a19f8089541e3845de3dc9a3ac9dd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.516616 20338 tablet_replica.cc:331] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd: stopping tablet replica
I20250626 02:03:34.516706 20338 raft_consensus.cc:2241] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.516803 20338 raft_consensus.cc:2270] T e37650dfb570400fb02742cb1220c7e1 P 145a19f8089541e3845de3dc9a3ac9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.529989 20338 tablet_server.cc:195] TabletServer@127.19.220.132:0 shutdown complete.
I20250626 02:03:34.532577 20338 tablet_server.cc:178] TabletServer@127.19.220.133:0 shutting down...
I20250626 02:03:34.537037 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:34.537205 20338 tablet_replica.cc:331] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8: stopping tablet replica
I20250626 02:03:34.537329 20338 raft_consensus.cc:2241] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.537465 20338 raft_consensus.cc:2270] T e37650dfb570400fb02742cb1220c7e1 P c5ee348337b146838a15e2d7161f5fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 02:03:34.537873 24171 consensus_peers.cc:487] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 -> Peer d150470bce35446b9b3f41ec78cb5064 (127.19.220.131:40133): Couldn't send request to peer d150470bce35446b9b3f41ec78cb5064. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.131:40133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:34.537899 20338 tablet_replica.cc:331] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8: stopping tablet replica
I20250626 02:03:34.538012 20338 raft_consensus.cc:2241] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.538105 20338 raft_consensus.cc:2270] T aa3a5817958042e8b2753a3df8e2ba4c P c5ee348337b146838a15e2d7161f5fb8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.538434 20338 tablet_replica.cc:331] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8: stopping tablet replica
I20250626 02:03:34.538534 20338 raft_consensus.cc:2241] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.538713 20338 raft_consensus.cc:2270] T 363d650f886244d38b78e6af7bb5f6ce P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.539012 20338 tablet_replica.cc:331] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8: stopping tablet replica
I20250626 02:03:34.539100 20338 raft_consensus.cc:2241] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.539201 20338 raft_consensus.cc:2270] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.539530 20338 tablet_replica.cc:331] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8: stopping tablet replica
I20250626 02:03:34.539615 20338 raft_consensus.cc:2241] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.539731 20338 raft_consensus.cc:2270] T 34ac2721a9bb44efa815122997c53fd7 P c5ee348337b146838a15e2d7161f5fb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.540004 20338 tablet_replica.cc:331] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8: stopping tablet replica
I20250626 02:03:34.540091 20338 raft_consensus.cc:2241] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 3 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.540252 20338 raft_consensus.cc:2270] T 65bf2a25991b4d4a94efbc900569a162 P c5ee348337b146838a15e2d7161f5fb8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.553351 20338 tablet_server.cc:195] TabletServer@127.19.220.133:0 shutdown complete.
I20250626 02:03:34.555974 20338 tablet_server.cc:178] TabletServer@127.19.220.134:0 shutting down...
I20250626 02:03:34.560307 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:34.560497 20338 tablet_replica.cc:331] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b: stopping tablet replica
I20250626 02:03:34.560622 20338 raft_consensus.cc:2241] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.560873 20338 raft_consensus.cc:2270] T 894851ab3c1e4d79971597e62b99cfbf P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.561305 20338 tablet_replica.cc:331] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b: stopping tablet replica
I20250626 02:03:34.561421 20338 raft_consensus.cc:2241] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.561555 20338 raft_consensus.cc:2270] T 363d650f886244d38b78e6af7bb5f6ce P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.561887 20338 tablet_replica.cc:331] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b: stopping tablet replica
I20250626 02:03:34.561978 20338 raft_consensus.cc:2241] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.562083 20338 raft_consensus.cc:2270] T 70698cc805a448b28cef88e47e146427 P 46dfe87116554377a3d4bf78c9f9699b [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.562386 20338 tablet_replica.cc:331] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b: stopping tablet replica
I20250626 02:03:34.562479 20338 raft_consensus.cc:2241] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:34.562594 20338 raft_consensus.cc:2270] T 13501c41bba94a7bbfb48a2715cd2d40 P 46dfe87116554377a3d4bf78c9f9699b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.562885 20338 tablet_replica.cc:331] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b: stopping tablet replica
I20250626 02:03:34.562965 20338 raft_consensus.cc:2241] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.563122 20338 raft_consensus.cc:2270] T 3a4a1bceeb5a4e3faba6c6ee9f5ec534 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.563444 20338 tablet_replica.cc:331] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b: stopping tablet replica
I20250626 02:03:34.563524 20338 raft_consensus.cc:2241] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.563658 20338 raft_consensus.cc:2270] T 34ac2721a9bb44efa815122997c53fd7 P 46dfe87116554377a3d4bf78c9f9699b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.576788 20338 tablet_server.cc:195] TabletServer@127.19.220.134:0 shutdown complete.
I20250626 02:03:34.579448 20338 master.cc:561] Master@127.19.220.190:42171 shutting down...
I20250626 02:03:34.583570 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:03:34.583737 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:34.583813 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P 37bae5288e804018b4dc9a3f7533d06c: stopping tablet replica
I20250626 02:03:34.598073 20338 master.cc:583] Master@127.19.220.190:42171 shutdown complete.
I20250626 02:03:34.601724 20338 test_util.cc:183] -----------------------------------------------
I20250626 02:03:34.601779 20338 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750903288526542-20338-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (35798 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20250626 02:03:34.602423 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:44209
I20250626 02:03:34.602744 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:34.604271 24870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:34.604353 24873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:34.604290 24871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:34.604645 20338 server_base.cc:1048] running on GCE node
I20250626 02:03:34.604782 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:34.604818 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:34.604842 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903414604841 us; error 0 us; skew 500 ppm
I20250626 02:03:34.605688 20338 webserver.cc:469] Webserver started at http://127.19.220.190:33705/ using document root <none> and password file <none>
I20250626 02:03:34.605834 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:34.605888 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:34.605965 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:34.606341 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "acb6f03e465c4eb9aca0f6ca699d9606"
format_stamp: "Formatted at 2025-06-26 02:03:34 on dist-test-slave-h18f"
I20250626 02:03:34.607708 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:34.608610 24878 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.608865 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:34.608932 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "acb6f03e465c4eb9aca0f6ca699d9606"
format_stamp: "Formatted at 2025-06-26 02:03:34 on dist-test-slave-h18f"
I20250626 02:03:34.609004 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:34.618839 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:34.619189 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:34.623147 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:44209
I20250626 02:03:34.632043 24930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:44209 every 8 connection(s)
I20250626 02:03:34.632572 24931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:34.634382 24931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606: Bootstrap starting.
I20250626 02:03:34.635161 24931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:34.636158 24931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606: No bootstrap required, opened a new log
I20250626 02:03:34.636531 24931 raft_consensus.cc:357] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f03e465c4eb9aca0f6ca699d9606" member_type: VOTER }
I20250626 02:03:34.636615 24931 raft_consensus.cc:383] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:34.636652 24931 raft_consensus.cc:738] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb6f03e465c4eb9aca0f6ca699d9606, State: Initialized, Role: FOLLOWER
I20250626 02:03:34.636793 24931 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f03e465c4eb9aca0f6ca699d9606" member_type: VOTER }
I20250626 02:03:34.636860 24931 raft_consensus.cc:397] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:03:34.636896 24931 raft_consensus.cc:491] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:03:34.636942 24931 raft_consensus.cc:3058] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:34.639176 24931 raft_consensus.cc:513] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f03e465c4eb9aca0f6ca699d9606" member_type: VOTER }
I20250626 02:03:34.639300 24931 leader_election.cc:304] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb6f03e465c4eb9aca0f6ca699d9606; no voters:
I20250626 02:03:34.639492 24931 leader_election.cc:290] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:03:34.639606 24934 raft_consensus.cc:2802] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:34.639813 24934 raft_consensus.cc:695] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 1 LEADER]: Becoming Leader. State: Replica: acb6f03e465c4eb9aca0f6ca699d9606, State: Running, Role: LEADER
I20250626 02:03:34.639933 24931 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:03:34.639945 24934 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f03e465c4eb9aca0f6ca699d9606" member_type: VOTER }
I20250626 02:03:34.640398 24935 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb6f03e465c4eb9aca0f6ca699d9606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f03e465c4eb9aca0f6ca699d9606" member_type: VOTER } }
I20250626 02:03:34.640424 24936 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb6f03e465c4eb9aca0f6ca699d9606. Latest consensus state: current_term: 1 leader_uuid: "acb6f03e465c4eb9aca0f6ca699d9606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb6f03e465c4eb9aca0f6ca699d9606" member_type: VOTER } }
I20250626 02:03:34.640617 24936 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [sys.catalog]: This master's current role is: LEADER
I20250626 02:03:34.640538 24935 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [sys.catalog]: This master's current role is: LEADER
I20250626 02:03:34.641085 24941 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:03:34.641798 24941 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:03:34.641996 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:03:34.643529 24941 catalog_manager.cc:1349] Generated new cluster ID: 21a199c00f134c91a43cf4b4cb1b1181
I20250626 02:03:34.643573 24941 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:03:34.656664 24941 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:03:34.657245 24941 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:03:34.663499 24941 catalog_manager.cc:5955] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606: Generated new TSK 0
I20250626 02:03:34.663667 24941 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:03:34.674365 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:34.676065 24952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:34.676157 24955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:34.676271 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:34.676366 24953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:34.676570 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:34.676613 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:34.676641 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903414676640 us; error 0 us; skew 500 ppm
I20250626 02:03:34.677580 20338 webserver.cc:469] Webserver started at http://127.19.220.129:37367/ using document root <none> and password file <none>
I20250626 02:03:34.677728 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:34.677783 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:34.677861 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:34.678244 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "8eda1e77519045ec9f9324c3ed129836"
format_stamp: "Formatted at 2025-06-26 02:03:34 on dist-test-slave-h18f"
I20250626 02:03:34.679672 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:03:34.680586 24960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.680792 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:34.680856 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "8eda1e77519045ec9f9324c3ed129836"
format_stamp: "Formatted at 2025-06-26 02:03:34 on dist-test-slave-h18f"
I20250626 02:03:34.680927 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:34.707130 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:34.707525 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:34.707834 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:34.708314 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:34.708354 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.708403 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:34.708444 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.712553 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:35909
I20250626 02:03:34.712585 25023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:35909 every 8 connection(s)
I20250626 02:03:34.713354 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:34.715242 25028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:34.715303 25029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:34.715423 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:34.715773 25031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:34.715997 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:34.716041 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:34.716075 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903414716074 us; error 0 us; skew 500 ppm
I20250626 02:03:34.717123 20338 webserver.cc:469] Webserver started at http://127.19.220.130:41867/ using document root <none> and password file <none>
I20250626 02:03:34.717298 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:34.717357 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:34.717517 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:34.717891 25024 heartbeater.cc:344] Connected to a master server at 127.19.220.190:44209
I20250626 02:03:34.717890 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "44404ab450604fd288d4bb2ef4cd61ca"
format_stamp: "Formatted at 2025-06-26 02:03:34 on dist-test-slave-h18f"
I20250626 02:03:34.718007 25024 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:34.718209 25024 heartbeater.cc:507] Master 127.19.220.190:44209 requested a full tablet report, sending...
I20250626 02:03:34.718760 24895 ts_manager.cc:194] Registered new tserver with Master: 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129:35909)
I20250626 02:03:34.719372 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:34.719426 24895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39730
I20250626 02:03:34.720319 25036 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.720541 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:34.720605 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "44404ab450604fd288d4bb2ef4cd61ca"
format_stamp: "Formatted at 2025-06-26 02:03:34 on dist-test-slave-h18f"
I20250626 02:03:34.720674 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:34.734784 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:34.735179 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:34.735497 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:34.735962 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:34.736002 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.736052 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:34.736085 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.740118 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:39613
I20250626 02:03:34.740177 25099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:39613 every 8 connection(s)
I20250626 02:03:34.741080 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:34.742825 25103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:34.742967 25104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:34.742889 25106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:34.743263 20338 server_base.cc:1048] running on GCE node
I20250626 02:03:34.743439 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:34.743486 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:34.743517 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903414743517 us; error 0 us; skew 500 ppm
I20250626 02:03:34.744789 20338 webserver.cc:469] Webserver started at http://127.19.220.131:34315/ using document root <none> and password file <none>
I20250626 02:03:34.744956 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:34.745016 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:34.745100 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:34.745151 25100 heartbeater.cc:344] Connected to a master server at 127.19.220.190:44209
I20250626 02:03:34.745246 25100 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:34.745478 25100 heartbeater.cc:507] Master 127.19.220.190:44209 requested a full tablet report, sending...
I20250626 02:03:34.745584 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "a95aceaff8de43b6966a82410ef61d9d"
format_stamp: "Formatted at 2025-06-26 02:03:34 on dist-test-slave-h18f"
I20250626 02:03:34.746093 24895 ts_manager.cc:194] Registered new tserver with Master: 44404ab450604fd288d4bb2ef4cd61ca (127.19.220.130:39613)
I20250626 02:03:34.746610 24895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39746
I20250626 02:03:34.747164 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:03:34.748057 25111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.748343 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:34.748401 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "a95aceaff8de43b6966a82410ef61d9d"
format_stamp: "Formatted at 2025-06-26 02:03:34 on dist-test-slave-h18f"
I20250626 02:03:34.748458 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:34.766172 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:34.766535 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:34.766795 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:34.767251 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:34.767289 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.767335 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:34.767365 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:34.771296 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:44841
I20250626 02:03:34.771348 25174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:44841 every 8 connection(s)
I20250626 02:03:34.775619 25175 heartbeater.cc:344] Connected to a master server at 127.19.220.190:44209
I20250626 02:03:34.775725 25175 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:34.775938 25175 heartbeater.cc:507] Master 127.19.220.190:44209 requested a full tablet report, sending...
I20250626 02:03:34.776500 24895 ts_manager.cc:194] Registered new tserver with Master: a95aceaff8de43b6966a82410ef61d9d (127.19.220.131:44841)
I20250626 02:03:34.776993 24895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39752
I20250626 02:03:34.777189 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005524511s
I20250626 02:03:35.720309 25024 heartbeater.cc:499] Master 127.19.220.190:44209 was elected leader, sending a full tablet report...
I20250626 02:03:35.747421 25100 heartbeater.cc:499] Master 127.19.220.190:44209 was elected leader, sending a full tablet report...
I20250626 02:03:35.777974 25175 heartbeater.cc:499] Master 127.19.220.190:44209 was elected leader, sending a full tablet report...
I20250626 02:03:35.803597 20338 test_util.cc:276] Using random seed: -131952951
I20250626 02:03:35.816305 24895 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39764:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:03:35.816836 24895 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:03:35.826778 24988 tablet_service.cc:1468] Processing CreateTablet for tablet 7966ca934ec74d0fafe72dd36a355eb2 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250626 02:03:35.826831 24987 tablet_service.cc:1468] Processing CreateTablet for tablet 5681b2d99802446a9af9ba42ff0ffc15 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250626 02:03:35.827111 24986 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa8533c431648869b444447e312fe21 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250626 02:03:35.827216 25139 tablet_service.cc:1468] Processing CreateTablet for tablet 7966ca934ec74d0fafe72dd36a355eb2 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250626 02:03:35.827507 25139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7966ca934ec74d0fafe72dd36a355eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.827688 24988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7966ca934ec74d0fafe72dd36a355eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.828568 25064 tablet_service.cc:1468] Processing CreateTablet for tablet 7966ca934ec74d0fafe72dd36a355eb2 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250626 02:03:35.828647 24987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5681b2d99802446a9af9ba42ff0ffc15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.828826 25064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7966ca934ec74d0fafe72dd36a355eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.830184 25138 tablet_service.cc:1468] Processing CreateTablet for tablet 5681b2d99802446a9af9ba42ff0ffc15 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250626 02:03:35.830441 25138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5681b2d99802446a9af9ba42ff0ffc15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.830500 25062 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa8533c431648869b444447e312fe21 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250626 02:03:35.830729 25062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa8533c431648869b444447e312fe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.830772 24986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa8533c431648869b444447e312fe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.832425 25063 tablet_service.cc:1468] Processing CreateTablet for tablet 5681b2d99802446a9af9ba42ff0ffc15 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250626 02:03:35.832691 25063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5681b2d99802446a9af9ba42ff0ffc15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.832681 25137 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa8533c431648869b444447e312fe21 (DEFAULT_TABLE table=test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250626 02:03:35.832923 25137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa8533c431648869b444447e312fe21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:35.833320 25197 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca: Bootstrap starting.
I20250626 02:03:35.833638 25195 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: Bootstrap starting.
I20250626 02:03:35.833709 25196 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836: Bootstrap starting.
I20250626 02:03:35.834800 25195 tablet_bootstrap.cc:654] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.835680 25197 tablet_bootstrap.cc:654] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.835829 25195 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: No bootstrap required, opened a new log
I20250626 02:03:35.835912 25195 ts_tablet_manager.cc:1397] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:03:35.836163 25196 tablet_bootstrap.cc:654] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.836325 25195 raft_consensus.cc:357] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.836553 25195 raft_consensus.cc:383] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.836633 25195 raft_consensus.cc:738] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95aceaff8de43b6966a82410ef61d9d, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.836771 25197 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca: No bootstrap required, opened a new log
I20250626 02:03:35.836807 25195 consensus_queue.cc:260] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.836858 25197 ts_tablet_manager.cc:1397] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250626 02:03:35.837170 25195 ts_tablet_manager.cc:1428] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:35.837316 25195 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d: Bootstrap starting.
I20250626 02:03:35.837294 25197 raft_consensus.cc:357] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.837370 25197 raft_consensus.cc:383] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.837397 25197 raft_consensus.cc:738] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44404ab450604fd288d4bb2ef4cd61ca, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.837494 25197 consensus_queue.cc:260] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.837566 25196 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836: No bootstrap required, opened a new log
I20250626 02:03:35.837672 25196 ts_tablet_manager.cc:1397] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:03:35.837808 25197 ts_tablet_manager.cc:1428] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:35.837975 25197 tablet_bootstrap.cc:492] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca: Bootstrap starting.
I20250626 02:03:35.838075 25196 raft_consensus.cc:357] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.838162 25196 raft_consensus.cc:383] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.838196 25196 raft_consensus.cc:738] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eda1e77519045ec9f9324c3ed129836, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.838284 25195 tablet_bootstrap.cc:654] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.838318 25196 consensus_queue.cc:260] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.838639 25196 ts_tablet_manager.cc:1428] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:35.838796 25196 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: Bootstrap starting.
I20250626 02:03:35.838918 25197 tablet_bootstrap.cc:654] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.839730 25196 tablet_bootstrap.cc:654] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.840016 25197 tablet_bootstrap.cc:492] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca: No bootstrap required, opened a new log
I20250626 02:03:35.840075 25197 ts_tablet_manager.cc:1397] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:03:35.840608 25197 raft_consensus.cc:357] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.840922 25197 raft_consensus.cc:383] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.840993 25196 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: No bootstrap required, opened a new log
I20250626 02:03:35.841003 25197 raft_consensus.cc:738] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44404ab450604fd288d4bb2ef4cd61ca, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.841091 25196 ts_tablet_manager.cc:1397] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:03:35.841166 25195 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d: No bootstrap required, opened a new log
I20250626 02:03:35.841233 25195 ts_tablet_manager.cc:1397] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:03:35.841205 25197 consensus_queue.cc:260] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.841509 25197 ts_tablet_manager.cc:1428] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:35.841491 25196 raft_consensus.cc:357] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.841560 25196 raft_consensus.cc:383] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.841580 25196 raft_consensus.cc:738] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eda1e77519045ec9f9324c3ed129836, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.841642 25195 raft_consensus.cc:357] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.841681 25196 consensus_queue.cc:260] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.841732 25195 raft_consensus.cc:383] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.841786 25195 raft_consensus.cc:738] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95aceaff8de43b6966a82410ef61d9d, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.841894 25195 consensus_queue.cc:260] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.841951 25196 ts_tablet_manager.cc:1428] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:35.842108 25196 tablet_bootstrap.cc:492] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836: Bootstrap starting.
I20250626 02:03:35.842108 25195 ts_tablet_manager.cc:1428] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:35.842332 25195 tablet_bootstrap.cc:492] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d: Bootstrap starting.
I20250626 02:03:35.842969 25196 tablet_bootstrap.cc:654] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.843284 25195 tablet_bootstrap.cc:654] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.841684 25197 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca: Bootstrap starting.
I20250626 02:03:35.844795 25197 tablet_bootstrap.cc:654] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:35.845052 25196 tablet_bootstrap.cc:492] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836: No bootstrap required, opened a new log
I20250626 02:03:35.845333 25196 ts_tablet_manager.cc:1397] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:03:35.845686 25196 raft_consensus.cc:357] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.845760 25196 raft_consensus.cc:383] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.845197 25195 tablet_bootstrap.cc:492] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d: No bootstrap required, opened a new log
I20250626 02:03:35.845882 25195 ts_tablet_manager.cc:1397] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:03:35.846334 25195 raft_consensus.cc:357] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.846419 25197 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca: No bootstrap required, opened a new log
I20250626 02:03:35.846438 25195 raft_consensus.cc:383] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.846470 25195 raft_consensus.cc:738] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95aceaff8de43b6966a82410ef61d9d, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.846500 25197 ts_tablet_manager.cc:1397] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 02:03:35.846593 25195 consensus_queue.cc:260] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.845794 25196 raft_consensus.cc:738] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eda1e77519045ec9f9324c3ed129836, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.846824 25196 consensus_queue.cc:260] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.846926 25197 raft_consensus.cc:357] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.847044 25197 raft_consensus.cc:383] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:35.847105 25197 raft_consensus.cc:738] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44404ab450604fd288d4bb2ef4cd61ca, State: Initialized, Role: FOLLOWER
I20250626 02:03:35.847110 25196 ts_tablet_manager.cc:1428] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:35.847324 25197 consensus_queue.cc:260] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.847597 25197 ts_tablet_manager.cc:1428] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:35.847757 25195 ts_tablet_manager.cc:1428] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:35.855998 25203 raft_consensus.cc:491] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:35.856117 25203 raft_consensus.cc:513] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.856635 25203 leader_election.cc:290] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44404ab450604fd288d4bb2ef4cd61ca (127.19.220.130:39613), a95aceaff8de43b6966a82410ef61d9d (127.19.220.131:44841)
I20250626 02:03:35.859835 25074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7966ca934ec74d0fafe72dd36a355eb2" candidate_uuid: "8eda1e77519045ec9f9324c3ed129836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44404ab450604fd288d4bb2ef4cd61ca" is_pre_election: true
I20250626 02:03:35.859877 25149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7966ca934ec74d0fafe72dd36a355eb2" candidate_uuid: "8eda1e77519045ec9f9324c3ed129836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95aceaff8de43b6966a82410ef61d9d" is_pre_election: true
I20250626 02:03:35.859975 25074 raft_consensus.cc:2466] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eda1e77519045ec9f9324c3ed129836 in term 0.
I20250626 02:03:35.859984 25149 raft_consensus.cc:2466] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eda1e77519045ec9f9324c3ed129836 in term 0.
I20250626 02:03:35.860271 24962 leader_election.cc:304] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44404ab450604fd288d4bb2ef4cd61ca, 8eda1e77519045ec9f9324c3ed129836; no voters:
I20250626 02:03:35.860427 25203 raft_consensus.cc:2802] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:35.860487 25203 raft_consensus.cc:491] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:35.860538 25203 raft_consensus.cc:3058] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.861378 25203 raft_consensus.cc:513] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.861742 25203 leader_election.cc:290] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [CANDIDATE]: Term 1 election: Requested vote from peers 44404ab450604fd288d4bb2ef4cd61ca (127.19.220.130:39613), a95aceaff8de43b6966a82410ef61d9d (127.19.220.131:44841)
I20250626 02:03:35.861968 25074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7966ca934ec74d0fafe72dd36a355eb2" candidate_uuid: "8eda1e77519045ec9f9324c3ed129836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44404ab450604fd288d4bb2ef4cd61ca"
I20250626 02:03:35.862056 25074 raft_consensus.cc:3058] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.862066 25149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7966ca934ec74d0fafe72dd36a355eb2" candidate_uuid: "8eda1e77519045ec9f9324c3ed129836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95aceaff8de43b6966a82410ef61d9d"
I20250626 02:03:35.862167 25149 raft_consensus.cc:3058] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.862452 25202 raft_consensus.cc:491] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:35.862541 25202 raft_consensus.cc:513] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.862958 25202 leader_election.cc:290] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129:35909), a95aceaff8de43b6966a82410ef61d9d (127.19.220.131:44841)
I20250626 02:03:35.862989 25149 raft_consensus.cc:2466] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eda1e77519045ec9f9324c3ed129836 in term 1.
I20250626 02:03:35.863116 25074 raft_consensus.cc:2466] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eda1e77519045ec9f9324c3ed129836 in term 1.
I20250626 02:03:35.863233 24963 leader_election.cc:304] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eda1e77519045ec9f9324c3ed129836, a95aceaff8de43b6966a82410ef61d9d; no voters:
I20250626 02:03:35.863469 25203 raft_consensus.cc:2802] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:35.863653 25203 raft_consensus.cc:695] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Becoming Leader. State: Replica: 8eda1e77519045ec9f9324c3ed129836, State: Running, Role: LEADER
I20250626 02:03:35.863796 25203 consensus_queue.cc:237] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } }
I20250626 02:03:35.865324 24895 catalog_manager.cc:5582] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "8eda1e77519045ec9f9324c3ed129836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:35.867364 25149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5681b2d99802446a9af9ba42ff0ffc15" candidate_uuid: "44404ab450604fd288d4bb2ef4cd61ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95aceaff8de43b6966a82410ef61d9d" is_pre_election: true
I20250626 02:03:35.867476 25149 raft_consensus.cc:2466] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44404ab450604fd288d4bb2ef4cd61ca in term 0.
I20250626 02:03:35.867502 24998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5681b2d99802446a9af9ba42ff0ffc15" candidate_uuid: "44404ab450604fd288d4bb2ef4cd61ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eda1e77519045ec9f9324c3ed129836" is_pre_election: true
I20250626 02:03:35.867617 24998 raft_consensus.cc:2466] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44404ab450604fd288d4bb2ef4cd61ca in term 0.
I20250626 02:03:35.867745 25039 leader_election.cc:304] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44404ab450604fd288d4bb2ef4cd61ca, a95aceaff8de43b6966a82410ef61d9d; no voters:
I20250626 02:03:35.867787 25203 raft_consensus.cc:491] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:35.867878 25203 raft_consensus.cc:513] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.868026 25202 raft_consensus.cc:2802] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:35.868088 25202 raft_consensus.cc:491] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:35.868149 25202 raft_consensus.cc:3058] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.868228 25203 leader_election.cc:290] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44404ab450604fd288d4bb2ef4cd61ca (127.19.220.130:39613), a95aceaff8de43b6966a82410ef61d9d (127.19.220.131:44841)
I20250626 02:03:35.868494 25149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8533c431648869b444447e312fe21" candidate_uuid: "8eda1e77519045ec9f9324c3ed129836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95aceaff8de43b6966a82410ef61d9d" is_pre_election: true
I20250626 02:03:35.868532 25074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8533c431648869b444447e312fe21" candidate_uuid: "8eda1e77519045ec9f9324c3ed129836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44404ab450604fd288d4bb2ef4cd61ca" is_pre_election: true
I20250626 02:03:35.868585 25149 raft_consensus.cc:2466] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eda1e77519045ec9f9324c3ed129836 in term 0.
I20250626 02:03:35.868609 25074 raft_consensus.cc:2466] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eda1e77519045ec9f9324c3ed129836 in term 0.
I20250626 02:03:35.868784 24963 leader_election.cc:304] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eda1e77519045ec9f9324c3ed129836, a95aceaff8de43b6966a82410ef61d9d; no voters:
I20250626 02:03:35.868980 25203 raft_consensus.cc:2802] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:35.869026 25203 raft_consensus.cc:491] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:35.869079 25203 raft_consensus.cc:3058] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.869131 25202 raft_consensus.cc:513] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.869410 25202 leader_election.cc:290] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [CANDIDATE]: Term 1 election: Requested vote from peers 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129:35909), a95aceaff8de43b6966a82410ef61d9d (127.19.220.131:44841)
I20250626 02:03:35.869710 24998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5681b2d99802446a9af9ba42ff0ffc15" candidate_uuid: "44404ab450604fd288d4bb2ef4cd61ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eda1e77519045ec9f9324c3ed129836"
I20250626 02:03:35.869858 24998 raft_consensus.cc:3058] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.869956 25203 raft_consensus.cc:513] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.870257 25203 leader_election.cc:290] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [CANDIDATE]: Term 1 election: Requested vote from peers 44404ab450604fd288d4bb2ef4cd61ca (127.19.220.130:39613), a95aceaff8de43b6966a82410ef61d9d (127.19.220.131:44841)
I20250626 02:03:35.870508 25074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8533c431648869b444447e312fe21" candidate_uuid: "8eda1e77519045ec9f9324c3ed129836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44404ab450604fd288d4bb2ef4cd61ca"
I20250626 02:03:35.870549 25148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8533c431648869b444447e312fe21" candidate_uuid: "8eda1e77519045ec9f9324c3ed129836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95aceaff8de43b6966a82410ef61d9d"
I20250626 02:03:35.870617 25148 raft_consensus.cc:3058] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.870617 25074 raft_consensus.cc:3058] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.870674 24998 raft_consensus.cc:2466] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44404ab450604fd288d4bb2ef4cd61ca in term 1.
I20250626 02:03:35.869709 25149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5681b2d99802446a9af9ba42ff0ffc15" candidate_uuid: "44404ab450604fd288d4bb2ef4cd61ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95aceaff8de43b6966a82410ef61d9d"
I20250626 02:03:35.870864 25149 raft_consensus.cc:3058] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:35.870973 25039 leader_election.cc:304] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44404ab450604fd288d4bb2ef4cd61ca, 8eda1e77519045ec9f9324c3ed129836; no voters:
I20250626 02:03:35.871121 25202 raft_consensus.cc:2802] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:35.871279 25202 raft_consensus.cc:695] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 LEADER]: Becoming Leader. State: Replica: 44404ab450604fd288d4bb2ef4cd61ca, State: Running, Role: LEADER
I20250626 02:03:35.871389 25202 consensus_queue.cc:237] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.871469 25148 raft_consensus.cc:2466] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eda1e77519045ec9f9324c3ed129836 in term 1.
I20250626 02:03:35.871469 25074 raft_consensus.cc:2466] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eda1e77519045ec9f9324c3ed129836 in term 1.
I20250626 02:03:35.871766 25149 raft_consensus.cc:2466] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44404ab450604fd288d4bb2ef4cd61ca in term 1.
I20250626 02:03:35.871927 24963 leader_election.cc:304] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eda1e77519045ec9f9324c3ed129836, a95aceaff8de43b6966a82410ef61d9d; no voters:
I20250626 02:03:35.872114 25203 raft_consensus.cc:2802] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:35.872218 25203 raft_consensus.cc:695] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Becoming Leader. State: Replica: 8eda1e77519045ec9f9324c3ed129836, State: Running, Role: LEADER
I20250626 02:03:35.872334 25203 consensus_queue.cc:237] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:35.873797 24895 catalog_manager.cc:5582] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 44404ab450604fd288d4bb2ef4cd61ca (127.19.220.130). New cstate: current_term: 1 leader_uuid: "44404ab450604fd288d4bb2ef4cd61ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:35.873823 24893 catalog_manager.cc:5582] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "8eda1e77519045ec9f9324c3ed129836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:35.885797 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:35.887630 25219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:35.887732 25220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:35.887732 25222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:35.888176 20338 server_base.cc:1048] running on GCE node
I20250626 02:03:35.888356 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:35.888393 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:35.888428 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903415888428 us; error 0 us; skew 500 ppm
I20250626 02:03:35.889442 20338 webserver.cc:469] Webserver started at http://127.19.220.132:45835/ using document root <none> and password file <none>
I20250626 02:03:35.889590 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:35.889642 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:35.889725 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:35.890091 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-3-root/instance:
uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43"
format_stamp: "Formatted at 2025-06-26 02:03:35 on dist-test-slave-h18f"
I20250626 02:03:35.891520 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:03:35.892423 25227 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:35.892684 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:35.892752 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-3-root
uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43"
format_stamp: "Formatted at 2025-06-26 02:03:35 on dist-test-slave-h18f"
I20250626 02:03:35.892828 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:35.898229 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:35.898613 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:35.898891 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:35.899312 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:35.899351 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:35.899398 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:35.899432 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:35.903877 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.132:36441
I20250626 02:03:35.903913 25290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.132:36441 every 8 connection(s)
I20250626 02:03:35.908214 25291 heartbeater.cc:344] Connected to a master server at 127.19.220.190:44209
I20250626 02:03:35.908311 25291 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:35.908531 25291 heartbeater.cc:507] Master 127.19.220.190:44209 requested a full tablet report, sending...
I20250626 02:03:35.909075 24893 ts_manager.cc:194] Registered new tserver with Master: a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132:36441)
I20250626 02:03:35.909595 24893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39772
I20250626 02:03:35.955188 25213 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 02:03:35.957227 25203 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250626 02:03:35.957297 25203 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250626 02:03:35.964272 25025 tablet.cc:2378] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 02:03:35.966132 25213 consensus_queue.cc:1035] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:35.977512 25213 consensus_queue.cc:1035] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:35.980515 25202 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:35.982147 25213 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:35.991163 25202 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:36.641953 24949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:36.646821 24998 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1aa8533c431648869b444447e312fe21"
dest_uuid: "8eda1e77519045ec9f9324c3ed129836"
mode: GRACEFUL
new_leader_uuid: "a95aceaff8de43b6966a82410ef61d9d"
from {username='slave'} at 127.0.0.1:34320
I20250626 02:03:36.646970 24998 raft_consensus.cc:604] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Received request to transfer leadership to TS a95aceaff8de43b6966a82410ef61d9d
I20250626 02:03:36.647253 24949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 02:03:36.647352 24949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:36.647534 25074 consensus_queue.cc:237] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } }
I20250626 02:03:36.648815 24998 raft_consensus.cc:1273] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Refusing update from remote peer 44404ab450604fd288d4bb2ef4cd61ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:36.649045 25149 raft_consensus.cc:1273] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Refusing update from remote peer 44404ab450604fd288d4bb2ef4cd61ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:36.649183 25311 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:03:36.649386 25304 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:03:36.650914 24998 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } } }
I20250626 02:03:36.650993 25149 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } } }
I20250626 02:03:36.652490 25311 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } } }
W20250626 02:03:36.653005 25040 consensus_peers.cc:487] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca -> Peer a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132:36441): Couldn't send request to peer a2d7047c87be4fdcbaa0cfb5ac48cd43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5681b2d99802446a9af9ba42ff0ffc15. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:36.653717 24895 catalog_manager.cc:5582] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d reported cstate change: config changed from index -1 to 2, NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "44404ab450604fd288d4bb2ef4cd61ca" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } } }
W20250626 02:03:36.658023 24943 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20250626 02:03:36.659539 24943 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1aa8533c431648869b444447e312fe21, TS a95aceaff8de43b6966a82410ef61d9d -> TS a2d7047c87be4fdcbaa0cfb5ac48cd43 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
W20250626 02:03:36.659597 24943 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 1aa8533c431648869b444447e312fe21, TS a95aceaff8de43b6966a82410ef61d9d -> TS a2d7047c87be4fdcbaa0cfb5ac48cd43 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:36.706359 25203 raft_consensus.cc:991] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836: : Instructing follower a95aceaff8de43b6966a82410ef61d9d to start an election
I20250626 02:03:36.706522 25203 raft_consensus.cc:1079] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Signalling peer a95aceaff8de43b6966a82410ef61d9d to start an election
I20250626 02:03:36.706934 25149 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1aa8533c431648869b444447e312fe21"
dest_uuid: "a95aceaff8de43b6966a82410ef61d9d"
from {username='slave'} at 127.0.0.1:44788
I20250626 02:03:36.707055 25149 raft_consensus.cc:491] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 02:03:36.707124 25149 raft_consensus.cc:3058] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:36.708263 25149 raft_consensus.cc:513] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:36.708742 25149 leader_election.cc:290] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [CANDIDATE]: Term 2 election: Requested vote from peers 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129:35909), 44404ab450604fd288d4bb2ef4cd61ca (127.19.220.130:39613)
I20250626 02:03:36.712345 24998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8533c431648869b444447e312fe21" candidate_uuid: "a95aceaff8de43b6966a82410ef61d9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8eda1e77519045ec9f9324c3ed129836"
I20250626 02:03:36.712472 24998 raft_consensus.cc:3053] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 02:03:36.712529 24998 raft_consensus.cc:738] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8eda1e77519045ec9f9324c3ed129836, State: Running, Role: LEADER
I20250626 02:03:36.712675 24998 consensus_queue.cc:260] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:36.712869 24998 raft_consensus.cc:3058] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:36.712906 25074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8533c431648869b444447e312fe21" candidate_uuid: "a95aceaff8de43b6966a82410ef61d9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44404ab450604fd288d4bb2ef4cd61ca"
I20250626 02:03:36.713049 25074 raft_consensus.cc:3058] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:03:36.713896 24998 raft_consensus.cc:2466] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a95aceaff8de43b6966a82410ef61d9d in term 2.
I20250626 02:03:36.713971 25074 raft_consensus.cc:2466] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a95aceaff8de43b6966a82410ef61d9d in term 2.
I20250626 02:03:36.714162 25114 leader_election.cc:304] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eda1e77519045ec9f9324c3ed129836, a95aceaff8de43b6966a82410ef61d9d; no voters:
I20250626 02:03:36.714316 25314 raft_consensus.cc:2802] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:03:36.714499 25314 raft_consensus.cc:695] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 2 LEADER]: Becoming Leader. State: Replica: a95aceaff8de43b6966a82410ef61d9d, State: Running, Role: LEADER
I20250626 02:03:36.714627 25314 consensus_queue.cc:237] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } }
I20250626 02:03:36.716019 24894 catalog_manager.cc:5582] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d reported cstate change: term changed from 1 to 2, leader changed from 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129) to a95aceaff8de43b6966a82410ef61d9d (127.19.220.131). New cstate: current_term: 2 leader_uuid: "a95aceaff8de43b6966a82410ef61d9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:36.740185 25323 ts_tablet_manager.cc:927] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Initiating tablet copy from peer 44404ab450604fd288d4bb2ef4cd61ca (127.19.220.130:39613)
I20250626 02:03:36.740638 25323 tablet_copy_client.cc:323] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:39613
I20250626 02:03:36.744665 25084 tablet_copy_service.cc:140] P 44404ab450604fd288d4bb2ef4cd61ca: Received BeginTabletCopySession request for tablet 5681b2d99802446a9af9ba42ff0ffc15 from peer a2d7047c87be4fdcbaa0cfb5ac48cd43 ({username='slave'} at 127.0.0.1:35360)
I20250626 02:03:36.744781 25084 tablet_copy_service.cc:161] P 44404ab450604fd288d4bb2ef4cd61ca: Beginning new tablet copy session on tablet 5681b2d99802446a9af9ba42ff0ffc15 from peer a2d7047c87be4fdcbaa0cfb5ac48cd43 at {username='slave'} at 127.0.0.1:35360: session id = a2d7047c87be4fdcbaa0cfb5ac48cd43-5681b2d99802446a9af9ba42ff0ffc15
I20250626 02:03:36.745741 25084 tablet_copy_source_session.cc:215] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:36.746512 25323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5681b2d99802446a9af9ba42ff0ffc15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:36.748430 25323 tablet_copy_client.cc:806] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:36.748617 25323 tablet_copy_client.cc:670] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:36.749648 25323 tablet_copy_client.cc:538] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:36.750801 25323 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Bootstrap starting.
I20250626 02:03:36.754305 25323 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:36.754535 25323 tablet_bootstrap.cc:492] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Bootstrap complete.
I20250626 02:03:36.754671 25323 ts_tablet_manager.cc:1397] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250626 02:03:36.755067 25323 raft_consensus.cc:357] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } }
I20250626 02:03:36.755178 25323 raft_consensus.cc:738] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2d7047c87be4fdcbaa0cfb5ac48cd43, State: Initialized, Role: LEARNER
I20250626 02:03:36.755304 25323 consensus_queue.cc:260] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } }
I20250626 02:03:36.755658 25291 heartbeater.cc:499] Master 127.19.220.190:44209 was elected leader, sending a full tablet report...
I20250626 02:03:36.755973 25323 ts_tablet_manager.cc:1428] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250626 02:03:36.756412 25084 tablet_copy_service.cc:342] P 44404ab450604fd288d4bb2ef4cd61ca: Request end of tablet copy session a2d7047c87be4fdcbaa0cfb5ac48cd43-5681b2d99802446a9af9ba42ff0ffc15 received from {username='slave'} at 127.0.0.1:35360
I20250626 02:03:36.756484 25084 tablet_copy_service.cc:434] P 44404ab450604fd288d4bb2ef4cd61ca: ending tablet copy session a2d7047c87be4fdcbaa0cfb5ac48cd43-5681b2d99802446a9af9ba42ff0ffc15 on tablet 5681b2d99802446a9af9ba42ff0ffc15 with peer a2d7047c87be4fdcbaa0cfb5ac48cd43
I20250626 02:03:36.791837 24998 raft_consensus.cc:1273] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 2 FOLLOWER]: Refusing update from remote peer a95aceaff8de43b6966a82410ef61d9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:03:36.792248 25314 consensus_queue.cc:1035] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:03:36.833947 25073 raft_consensus.cc:1273] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 2 FOLLOWER]: Refusing update from remote peer a95aceaff8de43b6966a82410ef61d9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:03:36.834316 25314 consensus_queue.cc:1035] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:03:36.860442 25265 raft_consensus.cc:1215] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 02:03:37.015369 25315 raft_consensus.cc:1062] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca: attempting to promote NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 to VOTER
I20250626 02:03:37.015847 25315 consensus_queue.cc:237] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } }
I20250626 02:03:37.017072 25265 raft_consensus.cc:1273] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 LEARNER]: Refusing update from remote peer 44404ab450604fd288d4bb2ef4cd61ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:03:37.017102 24998 raft_consensus.cc:1273] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Refusing update from remote peer 44404ab450604fd288d4bb2ef4cd61ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:03:37.017083 25149 raft_consensus.cc:1273] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Refusing update from remote peer 44404ab450604fd288d4bb2ef4cd61ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:03:37.017553 25304 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:03:37.017750 25315 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:03:37.017925 25310 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:03:37.020062 25265 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.020182 25149 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.022439 24893 catalog_manager.cc:5582] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 reported cstate change: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "44404ab450604fd288d4bb2ef4cd61ca" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.056790 25315 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.058760 24998 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } attrs { replace: true } } peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.081837 25074 consensus_queue.cc:237] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } }
I20250626 02:03:37.082923 25265 raft_consensus.cc:1273] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Refusing update from remote peer 44404ab450604fd288d4bb2ef4cd61ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:03:37.082990 25149 raft_consensus.cc:1273] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Refusing update from remote peer 44404ab450604fd288d4bb2ef4cd61ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:03:37.083304 25304 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:03:37.083508 25310 consensus_queue.cc:1035] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:03:37.085325 25149 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.085399 25265 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.087478 24892 catalog_manager.cc:5582] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 reported cstate change: config changed from index 3 to 4, VOTER 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "44404ab450604fd288d4bb2ef4cd61ca" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.092162 24986 tablet_service.cc:1515] Processing DeleteTablet for tablet 5681b2d99802446a9af9ba42ff0ffc15 with delete_type TABLET_DATA_TOMBSTONED (TS 8eda1e77519045ec9f9324c3ed129836 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34294
I20250626 02:03:37.092641 25334 tablet_replica.cc:331] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: stopping tablet replica
I20250626 02:03:37.092772 25334 raft_consensus.cc:2241] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:37.092893 25334 raft_consensus.cc:2270] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:37.093370 25334 ts_tablet_manager.cc:1905] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:37.095196 25334 ts_tablet_manager.cc:1918] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 02:03:37.095265 25334 log.cc:1199] T 5681b2d99802446a9af9ba42ff0ffc15 P 8eda1e77519045ec9f9324c3ed129836: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/5681b2d99802446a9af9ba42ff0ffc15
I20250626 02:03:37.095729 24881 catalog_manager.cc:4928] TS 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129:35909): tablet 5681b2d99802446a9af9ba42ff0ffc15 (table test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]) successfully deleted
I20250626 02:03:37.276230 25315 raft_consensus.cc:2953] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:37.277782 24880 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5681b2d99802446a9af9ba42ff0ffc15 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:38.280282 24998 consensus_queue.cc:237] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } }
I20250626 02:03:38.281761 25149 raft_consensus.cc:1273] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Refusing update from remote peer 8eda1e77519045ec9f9324c3ed129836: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:38.281767 25074 raft_consensus.cc:1273] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Refusing update from remote peer 8eda1e77519045ec9f9324c3ed129836: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:03:38.282187 25336 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:03:38.282426 25203 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:03:38.284379 25149 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } } }
I20250626 02:03:38.284492 25074 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } } }
W20250626 02:03:38.285655 24964 consensus_peers.cc:487] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 -> Peer a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132:36441): Couldn't send request to peer a2d7047c87be4fdcbaa0cfb5ac48cd43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7966ca934ec74d0fafe72dd36a355eb2. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:38.286582 24892 catalog_manager.cc:5582] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca reported cstate change: config changed from index -1 to 2, NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "8eda1e77519045ec9f9324c3ed129836" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } } }
I20250626 02:03:38.368155 25337 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } } }
I20250626 02:03:38.379406 25346 ts_tablet_manager.cc:927] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Initiating tablet copy from peer 8eda1e77519045ec9f9324c3ed129836 (127.19.220.129:35909)
I20250626 02:03:38.379782 25346 tablet_copy_client.cc:323] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:35909
I20250626 02:03:38.383626 25008 tablet_copy_service.cc:140] P 8eda1e77519045ec9f9324c3ed129836: Received BeginTabletCopySession request for tablet 7966ca934ec74d0fafe72dd36a355eb2 from peer a2d7047c87be4fdcbaa0cfb5ac48cd43 ({username='slave'} at 127.0.0.1:34350)
I20250626 02:03:38.383725 25008 tablet_copy_service.cc:161] P 8eda1e77519045ec9f9324c3ed129836: Beginning new tablet copy session on tablet 7966ca934ec74d0fafe72dd36a355eb2 from peer a2d7047c87be4fdcbaa0cfb5ac48cd43 at {username='slave'} at 127.0.0.1:34350: session id = a2d7047c87be4fdcbaa0cfb5ac48cd43-7966ca934ec74d0fafe72dd36a355eb2
I20250626 02:03:38.384757 25008 tablet_copy_source_session.cc:215] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:03:38.385557 25346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7966ca934ec74d0fafe72dd36a355eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:38.387820 25346 tablet_copy_client.cc:806] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: tablet copy: Starting download of 0 data blocks...
I20250626 02:03:38.388008 25346 tablet_copy_client.cc:670] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: tablet copy: Starting download of 1 WAL segments...
I20250626 02:03:38.388955 25346 tablet_copy_client.cc:538] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:03:38.389990 25346 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Bootstrap starting.
I20250626 02:03:38.393540 25346 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:03:38.393781 25346 tablet_bootstrap.cc:492] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Bootstrap complete.
I20250626 02:03:38.393986 25346 ts_tablet_manager.cc:1397] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20250626 02:03:38.394356 25346 raft_consensus.cc:357] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } }
I20250626 02:03:38.394469 25346 raft_consensus.cc:738] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2d7047c87be4fdcbaa0cfb5ac48cd43, State: Initialized, Role: LEARNER
I20250626 02:03:38.394587 25346 consensus_queue.cc:260] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: true } }
I20250626 02:03:38.394925 25346 ts_tablet_manager.cc:1428] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:38.395604 25008 tablet_copy_service.cc:342] P 8eda1e77519045ec9f9324c3ed129836: Request end of tablet copy session a2d7047c87be4fdcbaa0cfb5ac48cd43-7966ca934ec74d0fafe72dd36a355eb2 received from {username='slave'} at 127.0.0.1:34350
I20250626 02:03:38.395692 25008 tablet_copy_service.cc:434] P 8eda1e77519045ec9f9324c3ed129836: ending tablet copy session a2d7047c87be4fdcbaa0cfb5ac48cd43-7966ca934ec74d0fafe72dd36a355eb2 on tablet 7966ca934ec74d0fafe72dd36a355eb2 with peer a2d7047c87be4fdcbaa0cfb5ac48cd43
I20250626 02:03:38.491015 25265 raft_consensus.cc:1215] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 02:03:38.647682 24949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:38.648147 24949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:38.648239 24949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:38.649906 25203 raft_consensus.cc:1062] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836: attempting to promote NON_VOTER a2d7047c87be4fdcbaa0cfb5ac48cd43 to VOTER
I20250626 02:03:38.650285 25203 consensus_queue.cc:237] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } }
I20250626 02:03:38.651479 25265 raft_consensus.cc:1273] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 LEARNER]: Refusing update from remote peer 8eda1e77519045ec9f9324c3ed129836: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:03:38.651475 25074 raft_consensus.cc:1273] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Refusing update from remote peer 8eda1e77519045ec9f9324c3ed129836: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:03:38.651571 25149 raft_consensus.cc:1273] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Refusing update from remote peer 8eda1e77519045ec9f9324c3ed129836: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:03:38.651899 25352 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:03:38.652153 25336 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:03:38.652403 25337 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:03:38.654240 25149 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.654240 25265 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.656462 24895 catalog_manager.cc:5582] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d reported cstate change: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8eda1e77519045ec9f9324c3ed129836" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.729779 25074 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.744999 25203 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a2d7047c87be4fdcbaa0cfb5ac48cd43 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "a95aceaff8de43b6966a82410ef61d9d" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 44841 } attrs { replace: true } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.748061 24998 consensus_queue.cc:237] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } }
I20250626 02:03:38.749128 25265 raft_consensus.cc:1273] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Refusing update from remote peer 8eda1e77519045ec9f9324c3ed129836: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:03:38.749205 25074 raft_consensus.cc:1273] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Refusing update from remote peer 8eda1e77519045ec9f9324c3ed129836: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:03:38.749513 25354 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:03:38.749724 25337 consensus_queue.cc:1035] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:03:38.751333 25265 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a95aceaff8de43b6966a82410ef61d9d (127.19.220.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.751420 25074 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a95aceaff8de43b6966a82410ef61d9d (127.19.220.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.753587 24895 catalog_manager.cc:5582] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca reported cstate change: config changed from index 3 to 4, VOTER a95aceaff8de43b6966a82410ef61d9d (127.19.220.131) evicted. New cstate: current_term: 1 leader_uuid: "8eda1e77519045ec9f9324c3ed129836" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.779881 25137 tablet_service.cc:1515] Processing DeleteTablet for tablet 7966ca934ec74d0fafe72dd36a355eb2 with delete_type TABLET_DATA_TOMBSTONED (TS a95aceaff8de43b6966a82410ef61d9d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44784
I20250626 02:03:38.780328 25355 tablet_replica.cc:331] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: stopping tablet replica
I20250626 02:03:38.780660 25355 raft_consensus.cc:2241] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:38.780905 25355 raft_consensus.cc:2270] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:38.781718 25355 ts_tablet_manager.cc:1905] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:03:38.784049 25355 ts_tablet_manager.cc:1918] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 02:03:38.784121 25355 log.cc:1199] T 7966ca934ec74d0fafe72dd36a355eb2 P a95aceaff8de43b6966a82410ef61d9d: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/7966ca934ec74d0fafe72dd36a355eb2
I20250626 02:03:38.784775 24881 catalog_manager.cc:4928] TS a95aceaff8de43b6966a82410ef61d9d (127.19.220.131:44841): tablet 7966ca934ec74d0fafe72dd36a355eb2 (table test-workload [id=482d20aac64a43cabac4a3f76a2b1f3b]) successfully deleted
I20250626 02:03:38.905011 25203 raft_consensus.cc:2953] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a95aceaff8de43b6966a82410ef61d9d (127.19.220.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44404ab450604fd288d4bb2ef4cd61ca" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39613 } } peers { permanent_uuid: "8eda1e77519045ec9f9324c3ed129836" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 35909 } } peers { permanent_uuid: "a2d7047c87be4fdcbaa0cfb5ac48cd43" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 36441 } attrs { promote: false } } }
I20250626 02:03:38.906782 24881 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 7966ca934ec74d0fafe72dd36a355eb2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:03:40.648675 24949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:40.649174 24949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:40.649250 24949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:42.649474 24949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:42.649895 24949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:42.649951 24949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:44.650302 24949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:44.650727 24949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:44.650789 24949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:44.955327 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:03:44.958904 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:44.959089 20338 tablet_replica.cc:331] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836: stopping tablet replica
I20250626 02:03:44.959209 20338 raft_consensus.cc:2241] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:44.959316 20338 raft_consensus.cc:2270] T 1aa8533c431648869b444447e312fe21 P 8eda1e77519045ec9f9324c3ed129836 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:44.959785 20338 tablet_replica.cc:331] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836: stopping tablet replica
I20250626 02:03:44.959859 20338 raft_consensus.cc:2241] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:03:44.960019 20338 raft_consensus.cc:2270] T 7966ca934ec74d0fafe72dd36a355eb2 P 8eda1e77519045ec9f9324c3ed129836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:44.973033 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:03:44.975307 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:03:44.978749 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:44.978901 20338 tablet_replica.cc:331] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca: stopping tablet replica
I20250626 02:03:44.978992 20338 raft_consensus.cc:2241] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:03:44.979136 20338 raft_consensus.cc:2270] T 5681b2d99802446a9af9ba42ff0ffc15 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:44.979503 20338 tablet_replica.cc:331] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca: stopping tablet replica
I20250626 02:03:44.979580 20338 raft_consensus.cc:2241] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:44.979668 20338 raft_consensus.cc:2270] T 7966ca934ec74d0fafe72dd36a355eb2 P 44404ab450604fd288d4bb2ef4cd61ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:44.979934 20338 tablet_replica.cc:331] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca: stopping tablet replica
I20250626 02:03:44.979997 20338 raft_consensus.cc:2241] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:44.980062 20338 raft_consensus.cc:2270] T 1aa8533c431648869b444447e312fe21 P 44404ab450604fd288d4bb2ef4cd61ca [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:44.993005 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:03:44.995206 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:03:44.999048 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:44.999217 20338 tablet_replica.cc:331] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d: stopping tablet replica
I20250626 02:03:44.999327 20338 raft_consensus.cc:2241] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:03:44.999493 20338 raft_consensus.cc:2270] T 1aa8533c431648869b444447e312fe21 P a95aceaff8de43b6966a82410ef61d9d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:44.999883 20338 tablet_replica.cc:331] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d: stopping tablet replica
I20250626 02:03:44.999956 20338 raft_consensus.cc:2241] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:45.000047 20338 raft_consensus.cc:2270] T 5681b2d99802446a9af9ba42ff0ffc15 P a95aceaff8de43b6966a82410ef61d9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:45.012562 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:03:45.014714 20338 tablet_server.cc:178] TabletServer@127.19.220.132:0 shutting down...
I20250626 02:03:45.018038 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:45.018204 20338 tablet_replica.cc:331] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43: stopping tablet replica
I20250626 02:03:45.018314 20338 raft_consensus.cc:2241] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:45.018429 20338 raft_consensus.cc:2270] T 7966ca934ec74d0fafe72dd36a355eb2 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:45.018832 20338 tablet_replica.cc:331] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43: stopping tablet replica
I20250626 02:03:45.018899 20338 raft_consensus.cc:2241] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:45.018986 20338 raft_consensus.cc:2270] T 5681b2d99802446a9af9ba42ff0ffc15 P a2d7047c87be4fdcbaa0cfb5ac48cd43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:45.031241 20338 tablet_server.cc:195] TabletServer@127.19.220.132:0 shutdown complete.
I20250626 02:03:45.033231 20338 master.cc:561] Master@127.19.220.190:44209 shutting down...
I20250626 02:03:45.036887 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:03:45.037034 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:45.037102 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P acb6f03e465c4eb9aca0f6ca699d9606: stopping tablet replica
I20250626 02:03:45.049575 20338 master.cc:583] Master@127.19.220.190:44209 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10454 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20250626 02:03:45.057466 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:35833
I20250626 02:03:45.057818 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:45.059494 25362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:45.059664 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:45.059494 25364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:45.059602 25361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:45.059904 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:45.059947 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:45.059962 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903425059961 us; error 0 us; skew 500 ppm
I20250626 02:03:45.064388 20338 webserver.cc:469] Webserver started at http://127.19.220.190:38603/ using document root <none> and password file <none>
I20250626 02:03:45.064520 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:45.064563 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:45.064613 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:45.064994 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "fb648594f19a463f9a636907a270ef85"
format_stamp: "Formatted at 2025-06-26 02:03:45 on dist-test-slave-h18f"
I20250626 02:03:45.066386 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:03:45.067219 25369 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.067427 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:45.067493 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "fb648594f19a463f9a636907a270ef85"
format_stamp: "Formatted at 2025-06-26 02:03:45 on dist-test-slave-h18f"
I20250626 02:03:45.067559 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:45.079113 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:45.079427 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:45.083216 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:35833
I20250626 02:03:45.086338 25421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:35833 every 8 connection(s)
I20250626 02:03:45.086753 25422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:45.088680 25422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85: Bootstrap starting.
I20250626 02:03:45.089428 25422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:45.090343 25422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85: No bootstrap required, opened a new log
I20250626 02:03:45.090714 25422 raft_consensus.cc:357] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb648594f19a463f9a636907a270ef85" member_type: VOTER }
I20250626 02:03:45.090811 25422 raft_consensus.cc:383] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:45.090840 25422 raft_consensus.cc:738] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb648594f19a463f9a636907a270ef85, State: Initialized, Role: FOLLOWER
I20250626 02:03:45.090929 25422 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb648594f19a463f9a636907a270ef85" member_type: VOTER }
I20250626 02:03:45.090983 25422 raft_consensus.cc:397] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:03:45.091009 25422 raft_consensus.cc:491] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:03:45.091043 25422 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:45.091676 25422 raft_consensus.cc:513] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb648594f19a463f9a636907a270ef85" member_type: VOTER }
I20250626 02:03:45.091779 25422 leader_election.cc:304] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb648594f19a463f9a636907a270ef85; no voters:
I20250626 02:03:45.091912 25422 leader_election.cc:290] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:03:45.092024 25425 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:45.092218 25425 raft_consensus.cc:695] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 1 LEADER]: Becoming Leader. State: Replica: fb648594f19a463f9a636907a270ef85, State: Running, Role: LEADER
I20250626 02:03:45.092367 25425 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb648594f19a463f9a636907a270ef85" member_type: VOTER }
I20250626 02:03:45.092392 25422 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:03:45.092805 25426 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb648594f19a463f9a636907a270ef85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb648594f19a463f9a636907a270ef85" member_type: VOTER } }
I20250626 02:03:45.092850 25427 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb648594f19a463f9a636907a270ef85. Latest consensus state: current_term: 1 leader_uuid: "fb648594f19a463f9a636907a270ef85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb648594f19a463f9a636907a270ef85" member_type: VOTER } }
I20250626 02:03:45.092993 25427 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [sys.catalog]: This master's current role is: LEADER
I20250626 02:03:45.092975 25426 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [sys.catalog]: This master's current role is: LEADER
I20250626 02:03:45.093518 25431 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:03:45.094416 25431 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:03:45.094610 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:03:45.096177 25431 catalog_manager.cc:1349] Generated new cluster ID: bb9549f5827e4366965299e54586b837
I20250626 02:03:45.096238 25431 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:03:45.110634 25431 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:03:45.111225 25431 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:03:45.118402 25431 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85: Generated new TSK 0
I20250626 02:03:45.118593 25431 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:03:45.127019 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:45.128855 25446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:45.128927 25444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:45.128969 25443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:45.129009 20338 server_base.cc:1048] running on GCE node
I20250626 02:03:45.129283 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:45.129349 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:45.129380 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903425129379 us; error 0 us; skew 500 ppm
I20250626 02:03:45.130265 20338 webserver.cc:469] Webserver started at http://127.19.220.129:39435/ using document root <none> and password file <none>
I20250626 02:03:45.130425 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:45.130479 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:45.130559 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:45.130951 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "26f46ab76206430d90c17335d7d1e814"
format_stamp: "Formatted at 2025-06-26 02:03:45 on dist-test-slave-h18f"
I20250626 02:03:45.132362 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:45.133306 25451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.133525 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:45.133590 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "26f46ab76206430d90c17335d7d1e814"
format_stamp: "Formatted at 2025-06-26 02:03:45 on dist-test-slave-h18f"
I20250626 02:03:45.133658 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:45.142676 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:45.143026 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:45.143286 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:45.143712 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:45.143749 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.143797 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:45.143827 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.147670 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:34055
I20250626 02:03:45.147706 25514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:34055 every 8 connection(s)
I20250626 02:03:45.148451 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:45.150254 25519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:45.150259 25520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:45.150349 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:45.150481 25522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:45.150767 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:45.150822 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:45.150839 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903425150839 us; error 0 us; skew 500 ppm
I20250626 02:03:45.151865 20338 webserver.cc:469] Webserver started at http://127.19.220.130:42715/ using document root <none> and password file <none>
I20250626 02:03:45.152014 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:45.152069 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:45.152169 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:45.152616 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "993ebf63106540819b66dace4b158006"
format_stamp: "Formatted at 2025-06-26 02:03:45 on dist-test-slave-h18f"
I20250626 02:03:45.152688 25515 heartbeater.cc:344] Connected to a master server at 127.19.220.190:35833
I20250626 02:03:45.152761 25515 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:45.153000 25515 heartbeater.cc:507] Master 127.19.220.190:35833 requested a full tablet report, sending...
I20250626 02:03:45.153652 25386 ts_manager.cc:194] Registered new tserver with Master: 26f46ab76206430d90c17335d7d1e814 (127.19.220.129:34055)
I20250626 02:03:45.154367 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:03:45.154458 25386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48594
I20250626 02:03:45.155346 25527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.155560 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:45.155625 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "993ebf63106540819b66dace4b158006"
format_stamp: "Formatted at 2025-06-26 02:03:45 on dist-test-slave-h18f"
I20250626 02:03:45.155719 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:45.168210 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:45.168586 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:45.168875 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:45.169353 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:45.169391 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.169447 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:45.169481 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.173512 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:40053
I20250626 02:03:45.173566 25590 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:40053 every 8 connection(s)
I20250626 02:03:45.174270 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:45.176052 25595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:45.176059 25594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:45.176188 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:45.176925 25597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:45.177173 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:45.177225 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:45.177248 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903425177248 us; error 0 us; skew 500 ppm
I20250626 02:03:45.178179 20338 webserver.cc:469] Webserver started at http://127.19.220.131:43013/ using document root <none> and password file <none>
I20250626 02:03:45.178270 25591 heartbeater.cc:344] Connected to a master server at 127.19.220.190:35833
I20250626 02:03:45.178319 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:45.178382 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:45.178388 25591 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:45.178505 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:45.178651 25591 heartbeater.cc:507] Master 127.19.220.190:35833 requested a full tablet report, sending...
I20250626 02:03:45.178972 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "5538f05e6ec64a2ea2e7605ed2db9bec"
format_stamp: "Formatted at 2025-06-26 02:03:45 on dist-test-slave-h18f"
I20250626 02:03:45.179232 25386 ts_manager.cc:194] Registered new tserver with Master: 993ebf63106540819b66dace4b158006 (127.19.220.130:40053)
I20250626 02:03:45.179729 25386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48606
I20250626 02:03:45.180559 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:03:45.181433 25602 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.181658 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:45.181735 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "5538f05e6ec64a2ea2e7605ed2db9bec"
format_stamp: "Formatted at 2025-06-26 02:03:45 on dist-test-slave-h18f"
I20250626 02:03:45.181800 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:45.185719 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:45.186002 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:45.186231 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:45.186614 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:45.186653 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.186713 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:45.186741 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:45.190845 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:41963
I20250626 02:03:45.190899 25665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:41963 every 8 connection(s)
I20250626 02:03:45.194816 25666 heartbeater.cc:344] Connected to a master server at 127.19.220.190:35833
I20250626 02:03:45.194895 25666 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:45.195070 25666 heartbeater.cc:507] Master 127.19.220.190:35833 requested a full tablet report, sending...
I20250626 02:03:45.195550 25386 ts_manager.cc:194] Registered new tserver with Master: 5538f05e6ec64a2ea2e7605ed2db9bec (127.19.220.131:41963)
I20250626 02:03:45.195749 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004505292s
I20250626 02:03:45.196266 25386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48618
I20250626 02:03:46.155437 25515 heartbeater.cc:499] Master 127.19.220.190:35833 was elected leader, sending a full tablet report...
I20250626 02:03:46.180610 25591 heartbeater.cc:499] Master 127.19.220.190:35833 was elected leader, sending a full tablet report...
I20250626 02:03:46.197094 25666 heartbeater.cc:499] Master 127.19.220.190:35833 was elected leader, sending a full tablet report...
I20250626 02:03:46.221968 20338 test_util.cc:276] Using random seed: -121534580
I20250626 02:03:46.229617 25386 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48634:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:03:46.230083 25386 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:03:46.238685 25630 tablet_service.cc:1468] Processing CreateTablet for tablet b1d05078d8ed4fefa20d28198f803a0c (DEFAULT_TABLE table=test-workload [id=97c365839030444fa8e4ac487696b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:03:46.238992 25630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d05078d8ed4fefa20d28198f803a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:46.239409 25555 tablet_service.cc:1468] Processing CreateTablet for tablet b1d05078d8ed4fefa20d28198f803a0c (DEFAULT_TABLE table=test-workload [id=97c365839030444fa8e4ac487696b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:03:46.239660 25555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d05078d8ed4fefa20d28198f803a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:46.240484 25479 tablet_service.cc:1468] Processing CreateTablet for tablet b1d05078d8ed4fefa20d28198f803a0c (DEFAULT_TABLE table=test-workload [id=97c365839030444fa8e4ac487696b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:03:46.240739 25479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d05078d8ed4fefa20d28198f803a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:46.241348 25686 tablet_bootstrap.cc:492] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec: Bootstrap starting.
I20250626 02:03:46.241803 25687 tablet_bootstrap.cc:492] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006: Bootstrap starting.
I20250626 02:03:46.242251 25686 tablet_bootstrap.cc:654] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:46.242785 25687 tablet_bootstrap.cc:654] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:46.243191 25686 tablet_bootstrap.cc:492] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec: No bootstrap required, opened a new log
I20250626 02:03:46.243265 25686 ts_tablet_manager.cc:1397] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:46.243700 25686 raft_consensus.cc:357] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.243784 25686 raft_consensus.cc:383] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:46.243805 25686 raft_consensus.cc:738] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5538f05e6ec64a2ea2e7605ed2db9bec, State: Initialized, Role: FOLLOWER
I20250626 02:03:46.243978 25687 tablet_bootstrap.cc:492] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006: No bootstrap required, opened a new log
I20250626 02:03:46.243952 25686 consensus_queue.cc:260] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.244076 25687 ts_tablet_manager.cc:1397] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:03:46.244026 25689 tablet_bootstrap.cc:492] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814: Bootstrap starting.
I20250626 02:03:46.244490 25686 ts_tablet_manager.cc:1428] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:46.244535 25687 raft_consensus.cc:357] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.244692 25687 raft_consensus.cc:383] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:46.244746 25687 raft_consensus.cc:738] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993ebf63106540819b66dace4b158006, State: Initialized, Role: FOLLOWER
I20250626 02:03:46.244882 25687 consensus_queue.cc:260] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.245204 25689 tablet_bootstrap.cc:654] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:46.245510 25687 ts_tablet_manager.cc:1428] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:46.246601 25689 tablet_bootstrap.cc:492] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814: No bootstrap required, opened a new log
I20250626 02:03:46.246728 25689 ts_tablet_manager.cc:1397] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250626 02:03:46.247102 25689 raft_consensus.cc:357] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.247231 25689 raft_consensus.cc:383] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:46.247282 25689 raft_consensus.cc:738] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f46ab76206430d90c17335d7d1e814, State: Initialized, Role: FOLLOWER
I20250626 02:03:46.247417 25689 consensus_queue.cc:260] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.247761 25689 ts_tablet_manager.cc:1428] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:46.403494 25691 raft_consensus.cc:491] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:46.403656 25691 raft_consensus.cc:513] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.404189 25691 leader_election.cc:290] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f46ab76206430d90c17335d7d1e814 (127.19.220.129:34055), 993ebf63106540819b66dace4b158006 (127.19.220.130:40053)
I20250626 02:03:46.407613 25489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1d05078d8ed4fefa20d28198f803a0c" candidate_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f46ab76206430d90c17335d7d1e814" is_pre_election: true
I20250626 02:03:46.407774 25489 raft_consensus.cc:2466] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5538f05e6ec64a2ea2e7605ed2db9bec in term 0.
I20250626 02:03:46.407809 25565 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1d05078d8ed4fefa20d28198f803a0c" candidate_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993ebf63106540819b66dace4b158006" is_pre_election: true
I20250626 02:03:46.407935 25565 raft_consensus.cc:2466] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5538f05e6ec64a2ea2e7605ed2db9bec in term 0.
I20250626 02:03:46.408088 25603 leader_election.cc:304] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f46ab76206430d90c17335d7d1e814, 5538f05e6ec64a2ea2e7605ed2db9bec; no voters:
I20250626 02:03:46.408293 25691 raft_consensus.cc:2802] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:46.408355 25691 raft_consensus.cc:491] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:46.408427 25691 raft_consensus.cc:3058] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:46.409232 25691 raft_consensus.cc:513] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.409547 25691 leader_election.cc:290] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [CANDIDATE]: Term 1 election: Requested vote from peers 26f46ab76206430d90c17335d7d1e814 (127.19.220.129:34055), 993ebf63106540819b66dace4b158006 (127.19.220.130:40053)
I20250626 02:03:46.409826 25489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1d05078d8ed4fefa20d28198f803a0c" candidate_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f46ab76206430d90c17335d7d1e814"
I20250626 02:03:46.409930 25489 raft_consensus.cc:3058] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:46.409941 25565 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1d05078d8ed4fefa20d28198f803a0c" candidate_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993ebf63106540819b66dace4b158006"
I20250626 02:03:46.410071 25565 raft_consensus.cc:3058] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:46.410766 25489 raft_consensus.cc:2466] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5538f05e6ec64a2ea2e7605ed2db9bec in term 1.
I20250626 02:03:46.410894 25565 raft_consensus.cc:2466] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5538f05e6ec64a2ea2e7605ed2db9bec in term 1.
I20250626 02:03:46.411016 25603 leader_election.cc:304] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f46ab76206430d90c17335d7d1e814, 5538f05e6ec64a2ea2e7605ed2db9bec; no voters:
I20250626 02:03:46.411193 25691 raft_consensus.cc:2802] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:46.411427 25691 raft_consensus.cc:695] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 1 LEADER]: Becoming Leader. State: Replica: 5538f05e6ec64a2ea2e7605ed2db9bec, State: Running, Role: LEADER
I20250626 02:03:46.411566 25691 consensus_queue.cc:237] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } }
I20250626 02:03:46.412986 25386 catalog_manager.cc:5582] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec reported cstate change: term changed from 0 to 1, leader changed from <none> to 5538f05e6ec64a2ea2e7605ed2db9bec (127.19.220.131). New cstate: current_term: 1 leader_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5538f05e6ec64a2ea2e7605ed2db9bec" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f46ab76206430d90c17335d7d1e814" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:46.422614 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:03:46.425830 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:46.426081 20338 tablet_replica.cc:331] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814: stopping tablet replica
I20250626 02:03:46.426192 20338 raft_consensus.cc:2241] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:46.426301 20338 raft_consensus.cc:2270] T b1d05078d8ed4fefa20d28198f803a0c P 26f46ab76206430d90c17335d7d1e814 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:46.438917 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
W20250626 02:03:46.817711 25603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34055: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250626 02:03:46.818537 25603 consensus_peers.cc:487] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec -> Peer 26f46ab76206430d90c17335d7d1e814 (127.19.220.129:34055): Couldn't send request to peer 26f46ab76206430d90c17335d7d1e814. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:46.819216 25691 consensus_queue.cc:1035] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "993ebf63106540819b66dace4b158006" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 40053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:47.094318 25440 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:47.094594 25440 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:47.094666 25440 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:49.094976 25440 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:49.095248 25440 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:49.095295 25440 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 02:03:49.493183 25603 consensus_peers.cc:487] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec -> Peer 26f46ab76206430d90c17335d7d1e814 (127.19.220.129:34055): Couldn't send request to peer 26f46ab76206430d90c17335d7d1e814. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250626 02:03:51.095585 25440 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:51.095893 25440 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:51.095943 25440 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 02:03:52.177635 25603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34055: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250626 02:03:52.178303 25603 consensus_peers.cc:487] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec -> Peer 26f46ab76206430d90c17335d7d1e814 (127.19.220.129:34055): Couldn't send request to peer 26f46ab76206430d90c17335d7d1e814. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250626 02:03:53.096294 25440 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:53.096604 25440 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:53.096681 25440 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 02:03:54.713985 25603 consensus_peers.cc:487] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec -> Peer 26f46ab76206430d90c17335d7d1e814 (127.19.220.129:34055): Couldn't send request to peer 26f46ab76206430d90c17335d7d1e814. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250626 02:03:55.096954 25440 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:55.097270 25440 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:55.097335 25440 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 02:03:57.097687 25440 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 02:03:57.098033 25440 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 02:03:57.098095 25440 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 02:03:57.346906 25603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34055: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250626 02:03:57.347620 25603 consensus_peers.cc:487] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec -> Peer 26f46ab76206430d90c17335d7d1e814 (127.19.220.129:34055): Couldn't send request to peer 26f46ab76206430d90c17335d7d1e814. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:34055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250626 02:03:57.492954 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:03:57.496467 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:57.496671 20338 tablet_replica.cc:331] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006: stopping tablet replica
I20250626 02:03:57.496798 20338 raft_consensus.cc:2241] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:57.496933 20338 raft_consensus.cc:2270] T b1d05078d8ed4fefa20d28198f803a0c P 993ebf63106540819b66dace4b158006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:57.509640 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:03:57.511662 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:03:57.514882 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:57.515054 20338 tablet_replica.cc:331] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec: stopping tablet replica
I20250626 02:03:57.515166 20338 raft_consensus.cc:2241] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:03:57.515343 20338 raft_consensus.cc:2270] T b1d05078d8ed4fefa20d28198f803a0c P 5538f05e6ec64a2ea2e7605ed2db9bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:57.527823 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:03:57.529597 20338 master.cc:561] Master@127.19.220.190:35833 shutting down...
I20250626 02:03:57.532563 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:03:57.532707 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:57.532778 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P fb648594f19a463f9a636907a270ef85: stopping tablet replica
I20250626 02:03:57.545037 20338 master.cc:583] Master@127.19.220.190:35833 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12493 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20250626 02:03:57.550920 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:42933
I20250626 02:03:57.551226 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:57.552774 25732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:57.552788 25733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:57.552888 25735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:57.552901 20338 server_base.cc:1048] running on GCE node
I20250626 02:03:57.553162 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:57.553211 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:57.553242 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903437553241 us; error 0 us; skew 500 ppm
I20250626 02:03:57.559778 20338 webserver.cc:469] Webserver started at http://127.19.220.190:43697/ using document root <none> and password file <none>
I20250626 02:03:57.559934 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:57.559988 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:57.560070 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:57.560461 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "c8588d89d26f457681b10c470ede261c"
format_stamp: "Formatted at 2025-06-26 02:03:57 on dist-test-slave-h18f"
I20250626 02:03:57.561884 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:03:57.562752 25740 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.563014 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:57.563081 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "c8588d89d26f457681b10c470ede261c"
format_stamp: "Formatted at 2025-06-26 02:03:57 on dist-test-slave-h18f"
I20250626 02:03:57.563151 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:57.567711 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:57.568009 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:57.572052 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:42933
I20250626 02:03:57.574203 25792 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:42933 every 8 connection(s)
I20250626 02:03:57.574607 25793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:57.576350 25793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c: Bootstrap starting.
I20250626 02:03:57.577090 25793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:57.577967 25793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c: No bootstrap required, opened a new log
I20250626 02:03:57.578332 25793 raft_consensus.cc:357] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8588d89d26f457681b10c470ede261c" member_type: VOTER }
I20250626 02:03:57.578414 25793 raft_consensus.cc:383] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:57.578449 25793 raft_consensus.cc:738] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8588d89d26f457681b10c470ede261c, State: Initialized, Role: FOLLOWER
I20250626 02:03:57.578595 25793 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8588d89d26f457681b10c470ede261c" member_type: VOTER }
I20250626 02:03:57.578660 25793 raft_consensus.cc:397] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:03:57.578699 25793 raft_consensus.cc:491] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:03:57.578749 25793 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:57.579381 25793 raft_consensus.cc:513] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8588d89d26f457681b10c470ede261c" member_type: VOTER }
I20250626 02:03:57.579499 25793 leader_election.cc:304] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8588d89d26f457681b10c470ede261c; no voters:
I20250626 02:03:57.579672 25793 leader_election.cc:290] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:03:57.579754 25796 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:57.579952 25796 raft_consensus.cc:695] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 1 LEADER]: Becoming Leader. State: Replica: c8588d89d26f457681b10c470ede261c, State: Running, Role: LEADER
I20250626 02:03:57.580060 25793 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:03:57.580065 25796 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8588d89d26f457681b10c470ede261c" member_type: VOTER }
I20250626 02:03:57.580513 25797 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8588d89d26f457681b10c470ede261c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8588d89d26f457681b10c470ede261c" member_type: VOTER } }
I20250626 02:03:57.580549 25798 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8588d89d26f457681b10c470ede261c. Latest consensus state: current_term: 1 leader_uuid: "c8588d89d26f457681b10c470ede261c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8588d89d26f457681b10c470ede261c" member_type: VOTER } }
I20250626 02:03:57.580662 25797 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [sys.catalog]: This master's current role is: LEADER
I20250626 02:03:57.580677 25798 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [sys.catalog]: This master's current role is: LEADER
I20250626 02:03:57.581238 25802 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:03:57.582058 25802 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:03:57.582242 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:03:57.583658 25802 catalog_manager.cc:1349] Generated new cluster ID: 377ee060891d44c8bb4fbf09a408d292
I20250626 02:03:57.583714 25802 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:03:57.601775 25802 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:03:57.602274 25802 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:03:57.607722 25802 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c: Generated new TSK 0
I20250626 02:03:57.607859 25802 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:03:57.614414 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:57.616179 25815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:57.616317 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:57.616209 25814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:57.616266 25817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:57.616564 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:57.616609 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:57.616636 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903437616635 us; error 0 us; skew 500 ppm
I20250626 02:03:57.617491 20338 webserver.cc:469] Webserver started at http://127.19.220.129:46115/ using document root <none> and password file <none>
I20250626 02:03:57.617638 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:57.617691 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:57.617784 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:57.618142 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "5b1b94cebb2948a3ad24d59990ed8072"
format_stamp: "Formatted at 2025-06-26 02:03:57 on dist-test-slave-h18f"
I20250626 02:03:57.619460 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:03:57.620265 25822 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.620476 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:57.620535 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "5b1b94cebb2948a3ad24d59990ed8072"
format_stamp: "Formatted at 2025-06-26 02:03:57 on dist-test-slave-h18f"
I20250626 02:03:57.620609 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:57.629570 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:57.629868 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:57.630102 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:57.630508 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:57.630545 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.630591 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:57.630623 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.634186 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:46201
I20250626 02:03:57.634223 25885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:46201 every 8 connection(s)
I20250626 02:03:57.634899 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:57.636464 25890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:57.636488 25891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:57.636621 25893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:57.636822 20338 server_base.cc:1048] running on GCE node
I20250626 02:03:57.636997 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:57.637037 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:57.637063 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903437637062 us; error 0 us; skew 500 ppm
I20250626 02:03:57.638137 20338 webserver.cc:469] Webserver started at http://127.19.220.130:39169/ using document root <none> and password file <none>
I20250626 02:03:57.638315 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:57.638365 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:57.638468 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:57.638883 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "f275d2c9647047109d009a08b8f62cd5"
format_stamp: "Formatted at 2025-06-26 02:03:57 on dist-test-slave-h18f"
I20250626 02:03:57.639031 25886 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42933
I20250626 02:03:57.639113 25886 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:57.639324 25886 heartbeater.cc:507] Master 127.19.220.190:42933 requested a full tablet report, sending...
I20250626 02:03:57.639884 25757 ts_manager.cc:194] Registered new tserver with Master: 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201)
I20250626 02:03:57.640476 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:03:57.640642 25757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37510
I20250626 02:03:57.641420 25898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.641673 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:57.641739 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "f275d2c9647047109d009a08b8f62cd5"
format_stamp: "Formatted at 2025-06-26 02:03:57 on dist-test-slave-h18f"
I20250626 02:03:57.641808 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:57.651470 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:57.651767 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:57.652019 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:57.652463 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:57.652501 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.652549 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:57.652578 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.656289 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:39285
I20250626 02:03:57.656352 25961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:39285 every 8 connection(s)
I20250626 02:03:57.657050 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:03:57.658818 25966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:03:57.658955 25965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:57.659412 20338 server_base.cc:1048] running on GCE node
W20250626 02:03:57.659655 25968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:03:57.659943 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:03:57.659997 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:03:57.660027 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903437660026 us; error 0 us; skew 500 ppm
I20250626 02:03:57.661127 20338 webserver.cc:469] Webserver started at http://127.19.220.131:39629/ using document root <none> and password file <none>
I20250626 02:03:57.661301 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:03:57.661368 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:03:57.661456 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:03:57.661962 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "dac645f0cf5f4399819aea164f760471"
format_stamp: "Formatted at 2025-06-26 02:03:57 on dist-test-slave-h18f"
I20250626 02:03:57.662544 25962 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42933
I20250626 02:03:57.662654 25962 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:57.662896 25962 heartbeater.cc:507] Master 127.19.220.190:42933 requested a full tablet report, sending...
I20250626 02:03:57.663610 25757 ts_manager.cc:194] Registered new tserver with Master: f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285)
I20250626 02:03:57.663815 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250626 02:03:57.664299 25757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37526
I20250626 02:03:57.665246 25973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.665560 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:57.665670 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "dac645f0cf5f4399819aea164f760471"
format_stamp: "Formatted at 2025-06-26 02:03:57 on dist-test-slave-h18f"
I20250626 02:03:57.665745 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:03:57.671684 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:03:57.672026 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:03:57.672330 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:03:57.672771 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:03:57.672809 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.672857 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:03:57.672892 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:03:57.677220 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:37981
I20250626 02:03:57.677277 26036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:37981 every 8 connection(s)
I20250626 02:03:57.681684 26037 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42933
I20250626 02:03:57.681779 26037 heartbeater.cc:461] Registering TS with master...
I20250626 02:03:57.681964 26037 heartbeater.cc:507] Master 127.19.220.190:42933 requested a full tablet report, sending...
I20250626 02:03:57.682538 25757 ts_manager.cc:194] Registered new tserver with Master: dac645f0cf5f4399819aea164f760471 (127.19.220.131:37981)
I20250626 02:03:57.683036 25757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37540
I20250626 02:03:57.683199 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005581314s
I20250626 02:03:58.641548 25886 heartbeater.cc:499] Master 127.19.220.190:42933 was elected leader, sending a full tablet report...
I20250626 02:03:58.665246 25962 heartbeater.cc:499] Master 127.19.220.190:42933 was elected leader, sending a full tablet report...
I20250626 02:03:58.683940 26037 heartbeater.cc:499] Master 127.19.220.190:42933 was elected leader, sending a full tablet report...
I20250626 02:03:58.709436 20338 test_util.cc:276] Using random seed: -109047112
I20250626 02:03:58.717128 25757 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37552:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:03:58.717692 25757 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:03:58.728296 25850 tablet_service.cc:1468] Processing CreateTablet for tablet 6c2398a33bdf417ea40325c660368946 (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:03:58.728390 25848 tablet_service.cc:1468] Processing CreateTablet for tablet bf07004c4a034218a0912eafd06537fb (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:03:58.728590 25847 tablet_service.cc:1468] Processing CreateTablet for tablet a7c051f83b10430185079f2255397885 (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:03:58.728672 25850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2398a33bdf417ea40325c660368946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.728819 25848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf07004c4a034218a0912eafd06537fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.728389 25849 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac3e843d7ec4fb787aa86c0716e978d (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:03:58.729226 25849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac3e843d7ec4fb787aa86c0716e978d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.731303 26001 tablet_service.cc:1468] Processing CreateTablet for tablet 6c2398a33bdf417ea40325c660368946 (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:03:58.731401 25847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c051f83b10430185079f2255397885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.731386 26000 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac3e843d7ec4fb787aa86c0716e978d (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:03:58.731541 26001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2398a33bdf417ea40325c660368946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.731657 26000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac3e843d7ec4fb787aa86c0716e978d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.732391 25998 tablet_service.cc:1468] Processing CreateTablet for tablet a7c051f83b10430185079f2255397885 (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:03:58.732620 25998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c051f83b10430185079f2255397885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.733767 26058 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471: Bootstrap starting.
I20250626 02:03:58.734536 25999 tablet_service.cc:1468] Processing CreateTablet for tablet bf07004c4a034218a0912eafd06537fb (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:03:58.734656 26058 tablet_bootstrap.cc:654] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.734762 25999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf07004c4a034218a0912eafd06537fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.735641 26058 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471: No bootstrap required, opened a new log
I20250626 02:03:58.735723 26058 ts_tablet_manager.cc:1397] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:58.736136 26058 raft_consensus.cc:357] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.736274 26058 raft_consensus.cc:383] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.736310 26058 raft_consensus.cc:738] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac645f0cf5f4399819aea164f760471, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.736421 26058 consensus_queue.cc:260] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.736788 26058 ts_tablet_manager.cc:1428] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:58.736994 26058 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471: Bootstrap starting.
I20250626 02:03:58.737215 25926 tablet_service.cc:1468] Processing CreateTablet for tablet 6c2398a33bdf417ea40325c660368946 (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:03:58.737380 25923 tablet_service.cc:1468] Processing CreateTablet for tablet a7c051f83b10430185079f2255397885 (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:03:58.737437 25926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2398a33bdf417ea40325c660368946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.737628 25923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c051f83b10430185079f2255397885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.737715 25925 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac3e843d7ec4fb787aa86c0716e978d (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:03:58.737938 26058 tablet_bootstrap.cc:654] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.737962 25925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac3e843d7ec4fb787aa86c0716e978d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.739500 26062 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5: Bootstrap starting.
I20250626 02:03:58.739933 25924 tablet_service.cc:1468] Processing CreateTablet for tablet bf07004c4a034218a0912eafd06537fb (DEFAULT_TABLE table=test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:03:58.740226 25924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf07004c4a034218a0912eafd06537fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:03:58.740427 26058 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471: No bootstrap required, opened a new log
I20250626 02:03:58.740489 26062 tablet_bootstrap.cc:654] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.740530 26058 ts_tablet_manager.cc:1397] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250626 02:03:58.741003 26058 raft_consensus.cc:357] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.741103 26058 raft_consensus.cc:383] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.741143 26058 raft_consensus.cc:738] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac645f0cf5f4399819aea164f760471, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.741252 26058 consensus_queue.cc:260] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.741539 26058 ts_tablet_manager.cc:1428] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:58.741703 26062 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5: No bootstrap required, opened a new log
I20250626 02:03:58.741711 26058 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471: Bootstrap starting.
I20250626 02:03:58.741838 26062 ts_tablet_manager.cc:1397] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:03:58.742244 26062 raft_consensus.cc:357] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.742363 26062 raft_consensus.cc:383] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.742424 26062 raft_consensus.cc:738] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f275d2c9647047109d009a08b8f62cd5, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.742581 26057 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072: Bootstrap starting.
I20250626 02:03:58.742564 26062 consensus_queue.cc:260] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.742749 26058 tablet_bootstrap.cc:654] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.743033 26062 ts_tablet_manager.cc:1428] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:58.743494 26062 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5: Bootstrap starting.
I20250626 02:03:58.743752 26057 tablet_bootstrap.cc:654] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.744024 26058 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471: No bootstrap required, opened a new log
I20250626 02:03:58.744163 26058 ts_tablet_manager.cc:1397] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:03:58.744544 26058 raft_consensus.cc:357] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.744614 26062 tablet_bootstrap.cc:654] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.744648 26058 raft_consensus.cc:383] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.744705 26058 raft_consensus.cc:738] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac645f0cf5f4399819aea164f760471, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.744832 26058 consensus_queue.cc:260] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.745044 26058 ts_tablet_manager.cc:1428] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:58.745215 26058 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471: Bootstrap starting.
I20250626 02:03:58.745671 26062 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5: No bootstrap required, opened a new log
I20250626 02:03:58.745761 26062 ts_tablet_manager.cc:1397] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:03:58.746174 26058 tablet_bootstrap.cc:654] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.746174 26062 raft_consensus.cc:357] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.746825 26057 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072: No bootstrap required, opened a new log
I20250626 02:03:58.746836 26062 raft_consensus.cc:383] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.746912 26062 raft_consensus.cc:738] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f275d2c9647047109d009a08b8f62cd5, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.746913 26057 ts_tablet_manager.cc:1397] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250626 02:03:58.747092 26062 consensus_queue.cc:260] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.747238 26058 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471: No bootstrap required, opened a new log
I20250626 02:03:58.747418 26062 ts_tablet_manager.cc:1428] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:58.747555 26058 ts_tablet_manager.cc:1397] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:58.747452 26057 raft_consensus.cc:357] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.747614 26057 raft_consensus.cc:383] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.747654 26062 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5: Bootstrap starting.
I20250626 02:03:58.747653 26057 raft_consensus.cc:738] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1b94cebb2948a3ad24d59990ed8072, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.747825 26057 consensus_queue.cc:260] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.747977 26058 raft_consensus.cc:357] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.748072 26058 raft_consensus.cc:383] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.748108 26058 raft_consensus.cc:738] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac645f0cf5f4399819aea164f760471, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.748184 26057 ts_tablet_manager.cc:1428] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:03:58.748626 26062 tablet_bootstrap.cc:654] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.748257 26058 consensus_queue.cc:260] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.748373 26057 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072: Bootstrap starting.
I20250626 02:03:58.749615 26057 tablet_bootstrap.cc:654] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.750283 26058 ts_tablet_manager.cc:1428] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250626 02:03:58.750954 26057 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072: No bootstrap required, opened a new log
I20250626 02:03:58.751034 26057 ts_tablet_manager.cc:1397] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250626 02:03:58.751448 26057 raft_consensus.cc:357] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.751544 26057 raft_consensus.cc:383] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.751585 26057 raft_consensus.cc:738] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1b94cebb2948a3ad24d59990ed8072, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.751713 26057 consensus_queue.cc:260] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.751974 26057 ts_tablet_manager.cc:1428] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250626 02:03:58.752211 26057 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072: Bootstrap starting.
I20250626 02:03:58.752343 26062 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5: No bootstrap required, opened a new log
I20250626 02:03:58.752408 26062 ts_tablet_manager.cc:1397] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 02:03:58.752863 26062 raft_consensus.cc:357] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.752964 26062 raft_consensus.cc:383] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.753005 26062 raft_consensus.cc:738] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f275d2c9647047109d009a08b8f62cd5, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.753189 26057 tablet_bootstrap.cc:654] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.753129 26062 consensus_queue.cc:260] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.753448 26062 ts_tablet_manager.cc:1428] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:58.753628 26062 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5: Bootstrap starting.
I20250626 02:03:58.754617 26062 tablet_bootstrap.cc:654] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.755595 26057 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072: No bootstrap required, opened a new log
I20250626 02:03:58.755676 26057 ts_tablet_manager.cc:1397] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:03:58.756040 26057 raft_consensus.cc:357] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.756150 26057 raft_consensus.cc:383] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.756198 26057 raft_consensus.cc:738] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1b94cebb2948a3ad24d59990ed8072, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.756319 26057 consensus_queue.cc:260] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.756496 26062 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5: No bootstrap required, opened a new log
I20250626 02:03:58.756582 26057 ts_tablet_manager.cc:1428] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:58.756589 26062 ts_tablet_manager.cc:1397] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:03:58.756829 26057 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072: Bootstrap starting.
I20250626 02:03:58.757050 26062 raft_consensus.cc:357] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.757155 26062 raft_consensus.cc:383] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.757191 26062 raft_consensus.cc:738] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f275d2c9647047109d009a08b8f62cd5, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.757313 26062 consensus_queue.cc:260] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.757576 26062 ts_tablet_manager.cc:1428] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:03:58.757637 26057 tablet_bootstrap.cc:654] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072: Neither blocks nor log segments found. Creating new log.
I20250626 02:03:58.758664 26057 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072: No bootstrap required, opened a new log
I20250626 02:03:58.758762 26057 ts_tablet_manager.cc:1397] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:03:58.759150 26057 raft_consensus.cc:357] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.759250 26057 raft_consensus.cc:383] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:03:58.759289 26057 raft_consensus.cc:738] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1b94cebb2948a3ad24d59990ed8072, State: Initialized, Role: FOLLOWER
I20250626 02:03:58.759393 26057 consensus_queue.cc:260] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.759645 26057 ts_tablet_manager.cc:1428] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:03:58.910384 26060 raft_consensus.cc:491] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:58.910557 26060 raft_consensus.cc:513] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.911139 26060 leader_election.cc:290] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201), f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285)
I20250626 02:03:58.914701 25936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7c051f83b10430185079f2255397885" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f275d2c9647047109d009a08b8f62cd5" is_pre_election: true
I20250626 02:03:58.914709 25860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7c051f83b10430185079f2255397885" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b94cebb2948a3ad24d59990ed8072" is_pre_election: true
I20250626 02:03:58.914844 25936 raft_consensus.cc:2466] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 0.
I20250626 02:03:58.914868 25860 raft_consensus.cc:2466] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 0.
I20250626 02:03:58.915158 25976 leader_election.cc:304] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dac645f0cf5f4399819aea164f760471, f275d2c9647047109d009a08b8f62cd5; no voters:
I20250626 02:03:58.915335 26060 raft_consensus.cc:2802] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:58.915407 26060 raft_consensus.cc:491] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:58.915449 26060 raft_consensus.cc:3058] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.916323 26060 raft_consensus.cc:513] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.916633 26060 leader_election.cc:290] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 1 election: Requested vote from peers 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201), f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285)
I20250626 02:03:58.916975 25936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7c051f83b10430185079f2255397885" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f275d2c9647047109d009a08b8f62cd5"
I20250626 02:03:58.917081 25936 raft_consensus.cc:3058] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.917912 25936 raft_consensus.cc:2466] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 1.
I20250626 02:03:58.918092 25860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7c051f83b10430185079f2255397885" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b94cebb2948a3ad24d59990ed8072"
I20250626 02:03:58.918152 25976 leader_election.cc:304] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dac645f0cf5f4399819aea164f760471, f275d2c9647047109d009a08b8f62cd5; no voters:
I20250626 02:03:58.918192 25860 raft_consensus.cc:3058] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.918318 26060 raft_consensus.cc:2802] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:58.918529 26060 raft_consensus.cc:695] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Becoming Leader. State: Replica: dac645f0cf5f4399819aea164f760471, State: Running, Role: LEADER
I20250626 02:03:58.918673 26060 consensus_queue.cc:237] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.919039 25860 raft_consensus.cc:2466] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 1.
I20250626 02:03:58.920089 25757 catalog_manager.cc:5582] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 reported cstate change: term changed from 0 to 1, leader changed from <none> to dac645f0cf5f4399819aea164f760471 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:58.923691 26064 raft_consensus.cc:491] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:58.923794 26064 raft_consensus.cc:513] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.924216 26064 leader_election.cc:290] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201), dac645f0cf5f4399819aea164f760471 (127.19.220.131:37981)
I20250626 02:03:58.928535 26011 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf07004c4a034218a0912eafd06537fb" candidate_uuid: "f275d2c9647047109d009a08b8f62cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac645f0cf5f4399819aea164f760471" is_pre_election: true
I20250626 02:03:58.928679 26011 raft_consensus.cc:2466] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f275d2c9647047109d009a08b8f62cd5 in term 0.
I20250626 02:03:58.928884 25860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf07004c4a034218a0912eafd06537fb" candidate_uuid: "f275d2c9647047109d009a08b8f62cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b94cebb2948a3ad24d59990ed8072" is_pre_election: true
I20250626 02:03:58.928980 25899 leader_election.cc:304] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dac645f0cf5f4399819aea164f760471, f275d2c9647047109d009a08b8f62cd5; no voters:
I20250626 02:03:58.928987 25860 raft_consensus.cc:2466] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f275d2c9647047109d009a08b8f62cd5 in term 0.
I20250626 02:03:58.929154 26064 raft_consensus.cc:2802] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:58.929220 26064 raft_consensus.cc:491] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:58.929260 26064 raft_consensus.cc:3058] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.930011 26064 raft_consensus.cc:513] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.930266 26064 leader_election.cc:290] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201), dac645f0cf5f4399819aea164f760471 (127.19.220.131:37981)
I20250626 02:03:58.930532 26011 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf07004c4a034218a0912eafd06537fb" candidate_uuid: "f275d2c9647047109d009a08b8f62cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac645f0cf5f4399819aea164f760471"
I20250626 02:03:58.930542 25860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf07004c4a034218a0912eafd06537fb" candidate_uuid: "f275d2c9647047109d009a08b8f62cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b94cebb2948a3ad24d59990ed8072"
I20250626 02:03:58.930621 26011 raft_consensus.cc:3058] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.930652 25860 raft_consensus.cc:3058] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.931346 25860 raft_consensus.cc:2466] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f275d2c9647047109d009a08b8f62cd5 in term 1.
I20250626 02:03:58.931362 26011 raft_consensus.cc:2466] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f275d2c9647047109d009a08b8f62cd5 in term 1.
I20250626 02:03:58.931625 25900 leader_election.cc:304] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1b94cebb2948a3ad24d59990ed8072, f275d2c9647047109d009a08b8f62cd5; no voters:
I20250626 02:03:58.931784 26064 raft_consensus.cc:2802] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:58.931988 26064 raft_consensus.cc:695] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 1 LEADER]: Becoming Leader. State: Replica: f275d2c9647047109d009a08b8f62cd5, State: Running, Role: LEADER
I20250626 02:03:58.932108 26064 consensus_queue.cc:237] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.933615 25757 catalog_manager.cc:5582] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f275d2c9647047109d009a08b8f62cd5 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "f275d2c9647047109d009a08b8f62cd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:58.947552 26069 raft_consensus.cc:491] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:58.947685 26069 raft_consensus.cc:513] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.948168 26069 leader_election.cc:290] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285), dac645f0cf5f4399819aea164f760471 (127.19.220.131:37981)
I20250626 02:03:58.951545 25936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2398a33bdf417ea40325c660368946" candidate_uuid: "5b1b94cebb2948a3ad24d59990ed8072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f275d2c9647047109d009a08b8f62cd5" is_pre_election: true
I20250626 02:03:58.951545 26011 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2398a33bdf417ea40325c660368946" candidate_uuid: "5b1b94cebb2948a3ad24d59990ed8072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac645f0cf5f4399819aea164f760471" is_pre_election: true
I20250626 02:03:58.951685 25936 raft_consensus.cc:2466] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1b94cebb2948a3ad24d59990ed8072 in term 0.
I20250626 02:03:58.951735 26011 raft_consensus.cc:2466] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1b94cebb2948a3ad24d59990ed8072 in term 0.
I20250626 02:03:58.951962 25825 leader_election.cc:304] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1b94cebb2948a3ad24d59990ed8072, f275d2c9647047109d009a08b8f62cd5; no voters:
I20250626 02:03:58.952145 26069 raft_consensus.cc:2802] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:58.952212 26069 raft_consensus.cc:491] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:58.952242 26069 raft_consensus.cc:3058] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.953083 26069 raft_consensus.cc:513] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.953418 26069 leader_election.cc:290] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [CANDIDATE]: Term 1 election: Requested vote from peers f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285), dac645f0cf5f4399819aea164f760471 (127.19.220.131:37981)
I20250626 02:03:58.953709 25936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2398a33bdf417ea40325c660368946" candidate_uuid: "5b1b94cebb2948a3ad24d59990ed8072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f275d2c9647047109d009a08b8f62cd5"
I20250626 02:03:58.953719 26011 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2398a33bdf417ea40325c660368946" candidate_uuid: "5b1b94cebb2948a3ad24d59990ed8072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dac645f0cf5f4399819aea164f760471"
I20250626 02:03:58.953814 25936 raft_consensus.cc:3058] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.953840 26011 raft_consensus.cc:3058] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.954571 25936 raft_consensus.cc:2466] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1b94cebb2948a3ad24d59990ed8072 in term 1.
I20250626 02:03:58.954576 26011 raft_consensus.cc:2466] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1b94cebb2948a3ad24d59990ed8072 in term 1.
I20250626 02:03:58.954854 25825 leader_election.cc:304] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1b94cebb2948a3ad24d59990ed8072, f275d2c9647047109d009a08b8f62cd5; no voters:
I20250626 02:03:58.955003 26069 raft_consensus.cc:2802] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:58.955176 26069 raft_consensus.cc:695] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 LEADER]: Becoming Leader. State: Replica: 5b1b94cebb2948a3ad24d59990ed8072, State: Running, Role: LEADER
I20250626 02:03:58.955302 26069 consensus_queue.cc:237] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.956691 25757 catalog_manager.cc:5582] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "5b1b94cebb2948a3ad24d59990ed8072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: UNKNOWN } } }
I20250626 02:03:58.964329 26060 raft_consensus.cc:491] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:03:58.964417 26060 raft_consensus.cc:513] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.964754 26060 leader_election.cc:290] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201), f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285)
I20250626 02:03:58.965039 25860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac3e843d7ec4fb787aa86c0716e978d" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b94cebb2948a3ad24d59990ed8072" is_pre_election: true
I20250626 02:03:58.965044 25936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac3e843d7ec4fb787aa86c0716e978d" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f275d2c9647047109d009a08b8f62cd5" is_pre_election: true
I20250626 02:03:58.965157 25860 raft_consensus.cc:2466] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 0.
I20250626 02:03:58.965169 25936 raft_consensus.cc:2466] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 0.
I20250626 02:03:58.965405 25975 leader_election.cc:304] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b1b94cebb2948a3ad24d59990ed8072, dac645f0cf5f4399819aea164f760471; no voters:
I20250626 02:03:58.965581 26060 raft_consensus.cc:2802] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:03:58.965634 26060 raft_consensus.cc:491] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:03:58.965689 26060 raft_consensus.cc:3058] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.966449 26060 raft_consensus.cc:513] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.966782 26060 leader_election.cc:290] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 1 election: Requested vote from peers 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201), f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285)
I20250626 02:03:58.967041 25860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac3e843d7ec4fb787aa86c0716e978d" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1b94cebb2948a3ad24d59990ed8072"
I20250626 02:03:58.967041 25936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac3e843d7ec4fb787aa86c0716e978d" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f275d2c9647047109d009a08b8f62cd5"
I20250626 02:03:58.967175 25936 raft_consensus.cc:3058] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.967175 25860 raft_consensus.cc:3058] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:03:58.968086 25860 raft_consensus.cc:2466] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 1.
I20250626 02:03:58.968086 25936 raft_consensus.cc:2466] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 1.
I20250626 02:03:58.968400 25976 leader_election.cc:304] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dac645f0cf5f4399819aea164f760471, f275d2c9647047109d009a08b8f62cd5; no voters:
I20250626 02:03:58.968676 26060 raft_consensus.cc:2802] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:03:58.968804 26060 raft_consensus.cc:695] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Becoming Leader. State: Replica: dac645f0cf5f4399819aea164f760471, State: Running, Role: LEADER
I20250626 02:03:58.968950 26060 consensus_queue.cc:237] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:03:58.970389 25757 catalog_manager.cc:5582] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 reported cstate change: term changed from 0 to 1, leader changed from <none> to dac645f0cf5f4399819aea164f760471 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } }
I20250626 02:03:59.019785 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:03:59.023413 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:03:59.023669 20338 tablet_replica.cc:331] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072: stopping tablet replica
I20250626 02:03:59.023862 20338 raft_consensus.cc:2241] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:59.023993 20338 raft_consensus.cc:2270] T a7c051f83b10430185079f2255397885 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:59.024502 20338 tablet_replica.cc:331] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072: stopping tablet replica
I20250626 02:03:59.024607 20338 raft_consensus.cc:2241] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:59.024708 20338 raft_consensus.cc:2270] T 9ac3e843d7ec4fb787aa86c0716e978d P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:59.025072 20338 tablet_replica.cc:331] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072: stopping tablet replica
I20250626 02:03:59.025167 20338 raft_consensus.cc:2241] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:03:59.025291 20338 pending_rounds.cc:70] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072: Trying to abort 1 pending ops.
I20250626 02:03:59.025323 20338 pending_rounds.cc:77] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171700485962203136 op_type: NO_OP noop_request { }
I20250626 02:03:59.025405 20338 raft_consensus.cc:2887] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:03:59.025466 20338 raft_consensus.cc:2270] T 6c2398a33bdf417ea40325c660368946 P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:59.025956 20338 tablet_replica.cc:331] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072: stopping tablet replica
I20250626 02:03:59.026044 20338 raft_consensus.cc:2241] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:03:59.026130 20338 raft_consensus.cc:2270] T bf07004c4a034218a0912eafd06537fb P 5b1b94cebb2948a3ad24d59990ed8072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:03:59.039247 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
W20250626 02:03:59.313642 25975 consensus_peers.cc:487] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:59.339771 26064 consensus_queue.cc:1035] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 02:03:59.342955 25900 consensus_peers.cc:487] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:59.343394 26060 consensus_queue.cc:1035] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 02:03:59.522720 25975 consensus_peers.cc:487] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:03:59.563608 26077 consensus_queue.cc:1035] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:03:59.933424 26091 consensus_queue.cc:579] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Leader has been unable to successfully communicate with peer 5b1b94cebb2948a3ad24d59990ed8072 for more than 1 seconds (1.001s)
W20250626 02:03:59.935420 25757 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf07004c4a034218a0912eafd06537fb with cas_config_opid_index -1: no extra replica candidate found for tablet bf07004c4a034218a0912eafd06537fb (table test-workload [id=99e8cfe85ef7428c8e787757dded0f9b]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250626 02:03:59.994868 26060 consensus_queue.cc:579] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Leader has been unable to successfully communicate with peer 5b1b94cebb2948a3ad24d59990ed8072 for more than 1 seconds (1.026s)
I20250626 02:04:00.042199 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:00.044222 26102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:00.044224 26104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:00.044258 26101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:00.044361 20338 server_base.cc:1048] running on GCE node
I20250626 02:04:00.044760 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:00.044807 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:00.044827 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903440044825 us; error 0 us; skew 500 ppm
I20250626 02:04:00.045884 20338 webserver.cc:469] Webserver started at http://127.19.220.132:44529/ using document root <none> and password file <none>
I20250626 02:04:00.046054 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:00.046123 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:00.046200 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:00.046581 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-3-root/instance:
uuid: "6da28d4a1cd14deb8a8e44f7bc50a162"
format_stamp: "Formatted at 2025-06-26 02:04:00 on dist-test-slave-h18f"
I20250626 02:04:00.048048 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:00.049007 26109 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:00.049255 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:00.049316 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-3-root
uuid: "6da28d4a1cd14deb8a8e44f7bc50a162"
format_stamp: "Formatted at 2025-06-26 02:04:00 on dist-test-slave-h18f"
I20250626 02:04:00.049371 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750903288526542-20338-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:00.062256 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:00.062561 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:00.062804 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:00.063225 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:00.063261 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:00.063293 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:00.063325 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:00.067416 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.132:34013
I20250626 02:04:00.067948 26172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.132:34013 every 8 connection(s)
I20250626 02:04:00.071743 26173 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42933
I20250626 02:04:00.071823 26173 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:00.072026 26173 heartbeater.cc:507] Master 127.19.220.190:42933 requested a full tablet report, sending...
I20250626 02:04:00.072646 25757 ts_manager.cc:194] Registered new tserver with Master: 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132:34013)
I20250626 02:04:00.073144 25757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37562
I20250626 02:04:00.101634 26011 consensus_queue.cc:237] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.103116 25936 raft_consensus.cc:1273] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:04:00.103510 26077 consensus_queue.cc:1035] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 02:04:00.104018 25975 consensus_peers.cc:487] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:00.104952 26095 raft_consensus.cc:2953] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } } }
I20250626 02:04:00.105332 25936 raft_consensus.cc:2953] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } } }
I20250626 02:04:00.106542 25741 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ac3e843d7ec4fb787aa86c0716e978d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
W20250626 02:04:00.107018 25974 consensus_peers.cc:487] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 -> Peer 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132:34013): Couldn't send request to peer 6da28d4a1cd14deb8a8e44f7bc50a162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ac3e843d7ec4fb787aa86c0716e978d. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:00.107427 25757 catalog_manager.cc:5582] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 reported cstate change: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:00.122757 25936 consensus_queue.cc:237] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.124466 26011 raft_consensus.cc:1273] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Refusing update from remote peer f275d2c9647047109d009a08b8f62cd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 02:04:00.124648 25900 consensus_peers.cc:487] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:00.124815 26091 consensus_queue.cc:1035] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:04:00.126281 26098 raft_consensus.cc:2953] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } } }
I20250626 02:04:00.126734 26011 raft_consensus.cc:2953] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } } }
W20250626 02:04:00.128264 25899 consensus_peers.cc:487] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 -> Peer 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132:34013): Couldn't send request to peer 6da28d4a1cd14deb8a8e44f7bc50a162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf07004c4a034218a0912eafd06537fb. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:00.128453 25743 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf07004c4a034218a0912eafd06537fb with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250626 02:04:00.129038 25756 catalog_manager.cc:5582] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 reported cstate change: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "f275d2c9647047109d009a08b8f62cd5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:00.316285 26077 consensus_queue.cc:579] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Leader has been unable to successfully communicate with peer 5b1b94cebb2948a3ad24d59990ed8072 for more than 1 seconds (1.397s)
I20250626 02:04:00.318653 26011 consensus_queue.cc:237] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.319859 25936 raft_consensus.cc:1273] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 02:04:00.320029 25974 consensus_peers.cc:487] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 -> Peer 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132:34013): Couldn't send request to peer 6da28d4a1cd14deb8a8e44f7bc50a162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7c051f83b10430185079f2255397885. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:00.320294 26095 consensus_queue.cc:1035] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 02:04:00.320442 25975 consensus_peers.cc:487] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:00.321578 26077 raft_consensus.cc:2953] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } } }
I20250626 02:04:00.321976 25936 raft_consensus.cc:2953] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } } }
I20250626 02:04:00.322942 25741 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7c051f83b10430185079f2255397885 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 02:04:00.323853 25757 catalog_manager.cc:5582] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 reported cstate change: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:00.521867 26184 raft_consensus.cc:491] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:00.522027 26184 raft_consensus.cc:513] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:04:00.522431 26184 leader_election.cc:290] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201), f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285)
I20250626 02:04:00.522905 25936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2398a33bdf417ea40325c660368946" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f275d2c9647047109d009a08b8f62cd5" is_pre_election: true
I20250626 02:04:00.523051 25936 raft_consensus.cc:2466] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 1.
I20250626 02:04:00.523351 25976 leader_election.cc:304] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dac645f0cf5f4399819aea164f760471, f275d2c9647047109d009a08b8f62cd5; no voters:
I20250626 02:04:00.523530 26184 raft_consensus.cc:2802] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250626 02:04:00.523553 25975 leader_election.cc:336] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
I20250626 02:04:00.523627 26184 raft_consensus.cc:491] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:00.523669 26184 raft_consensus.cc:3058] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:00.524642 26184 raft_consensus.cc:513] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:04:00.524951 26184 leader_election.cc:290] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 2 election: Requested vote from peers 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201), f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285)
I20250626 02:04:00.525290 25936 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c2398a33bdf417ea40325c660368946" candidate_uuid: "dac645f0cf5f4399819aea164f760471" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f275d2c9647047109d009a08b8f62cd5"
I20250626 02:04:00.525420 25936 raft_consensus.cc:3058] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Advancing to term 2
W20250626 02:04:00.525818 25975 leader_election.cc:336] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
I20250626 02:04:00.525995 26185 ts_tablet_manager.cc:927] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: Initiating tablet copy from peer f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285)
I20250626 02:04:00.526341 26185 tablet_copy_client.cc:323] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:39285
I20250626 02:04:00.526448 25936 raft_consensus.cc:2466] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac645f0cf5f4399819aea164f760471 in term 2.
I20250626 02:04:00.526698 25976 leader_election.cc:304] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dac645f0cf5f4399819aea164f760471, f275d2c9647047109d009a08b8f62cd5; no voters: 5b1b94cebb2948a3ad24d59990ed8072
I20250626 02:04:00.526859 26184 raft_consensus.cc:2802] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:04:00.527096 26184 raft_consensus.cc:695] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 2 LEADER]: Becoming Leader. State: Replica: dac645f0cf5f4399819aea164f760471, State: Running, Role: LEADER
I20250626 02:04:00.527264 26184 consensus_queue.cc:237] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } }
I20250626 02:04:00.528813 25756 catalog_manager.cc:5582] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 reported cstate change: term changed from 1 to 2, leader changed from 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) to dac645f0cf5f4399819aea164f760471 (127.19.220.131). New cstate: current_term: 2 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:00.531080 25946 tablet_copy_service.cc:140] P f275d2c9647047109d009a08b8f62cd5: Received BeginTabletCopySession request for tablet bf07004c4a034218a0912eafd06537fb from peer 6da28d4a1cd14deb8a8e44f7bc50a162 ({username='slave'} at 127.0.0.1:47372)
I20250626 02:04:00.531167 25946 tablet_copy_service.cc:161] P f275d2c9647047109d009a08b8f62cd5: Beginning new tablet copy session on tablet bf07004c4a034218a0912eafd06537fb from peer 6da28d4a1cd14deb8a8e44f7bc50a162 at {username='slave'} at 127.0.0.1:47372: session id = 6da28d4a1cd14deb8a8e44f7bc50a162-bf07004c4a034218a0912eafd06537fb
I20250626 02:04:00.531924 25946 tablet_copy_source_session.cc:215] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:00.532634 26185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf07004c4a034218a0912eafd06537fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:00.534305 26185 tablet_copy_client.cc:806] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:00.534458 26185 tablet_copy_client.cc:670] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:00.535374 26185 tablet_copy_client.cc:538] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:00.536388 26185 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap starting.
I20250626 02:04:00.539278 26185 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:00.539503 26185 tablet_bootstrap.cc:492] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap complete.
I20250626 02:04:00.539656 26185 ts_tablet_manager.cc:1397] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:00.540001 26185 raft_consensus.cc:357] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.540095 26185 raft_consensus.cc:738] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6da28d4a1cd14deb8a8e44f7bc50a162, State: Initialized, Role: LEARNER
I20250626 02:04:00.540220 26185 consensus_queue.cc:260] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.540513 26185 ts_tablet_manager.cc:1428] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:00.540616 26173 heartbeater.cc:499] Master 127.19.220.190:42933 was elected leader, sending a full tablet report...
I20250626 02:04:00.540982 25946 tablet_copy_service.cc:342] P f275d2c9647047109d009a08b8f62cd5: Request end of tablet copy session 6da28d4a1cd14deb8a8e44f7bc50a162-bf07004c4a034218a0912eafd06537fb received from {username='slave'} at 127.0.0.1:47372
I20250626 02:04:00.541062 25946 tablet_copy_service.cc:434] P f275d2c9647047109d009a08b8f62cd5: ending tablet copy session 6da28d4a1cd14deb8a8e44f7bc50a162-bf07004c4a034218a0912eafd06537fb on tablet bf07004c4a034218a0912eafd06537fb with peer 6da28d4a1cd14deb8a8e44f7bc50a162
W20250626 02:04:00.583527 25805 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 5b1b94cebb2948a3ad24d59990ed8072 not available for placement
I20250626 02:04:00.691821 26185 ts_tablet_manager.cc:927] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: Initiating tablet copy from peer dac645f0cf5f4399819aea164f760471 (127.19.220.131:37981)
I20250626 02:04:00.692189 26185 tablet_copy_client.cc:323] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:37981
I20250626 02:04:00.695408 26021 tablet_copy_service.cc:140] P dac645f0cf5f4399819aea164f760471: Received BeginTabletCopySession request for tablet 9ac3e843d7ec4fb787aa86c0716e978d from peer 6da28d4a1cd14deb8a8e44f7bc50a162 ({username='slave'} at 127.0.0.1:33010)
I20250626 02:04:00.695524 26021 tablet_copy_service.cc:161] P dac645f0cf5f4399819aea164f760471: Beginning new tablet copy session on tablet 9ac3e843d7ec4fb787aa86c0716e978d from peer 6da28d4a1cd14deb8a8e44f7bc50a162 at {username='slave'} at 127.0.0.1:33010: session id = 6da28d4a1cd14deb8a8e44f7bc50a162-9ac3e843d7ec4fb787aa86c0716e978d
I20250626 02:04:00.696337 26021 tablet_copy_source_session.cc:215] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:00.697163 26185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac3e843d7ec4fb787aa86c0716e978d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:00.698953 26185 tablet_copy_client.cc:806] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:00.699105 26185 tablet_copy_client.cc:670] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:00.700165 26185 tablet_copy_client.cc:538] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:00.701193 26185 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap starting.
I20250626 02:04:00.703910 26185 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:00.704144 26185 tablet_bootstrap.cc:492] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap complete.
I20250626 02:04:00.704298 26185 ts_tablet_manager.cc:1397] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: Time spent bootstrapping tablet: real 0.003s user 0.004s sys 0.000s
I20250626 02:04:00.704648 26185 raft_consensus.cc:357] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.704753 26185 raft_consensus.cc:738] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6da28d4a1cd14deb8a8e44f7bc50a162, State: Initialized, Role: LEARNER
I20250626 02:04:00.704869 26185 consensus_queue.cc:260] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.705090 26185 ts_tablet_manager.cc:1428] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:00.705585 26021 tablet_copy_service.cc:342] P dac645f0cf5f4399819aea164f760471: Request end of tablet copy session 6da28d4a1cd14deb8a8e44f7bc50a162-9ac3e843d7ec4fb787aa86c0716e978d received from {username='slave'} at 127.0.0.1:33010
I20250626 02:04:00.705670 26021 tablet_copy_service.cc:434] P dac645f0cf5f4399819aea164f760471: ending tablet copy session 6da28d4a1cd14deb8a8e44f7bc50a162-9ac3e843d7ec4fb787aa86c0716e978d on tablet 9ac3e843d7ec4fb787aa86c0716e978d with peer 6da28d4a1cd14deb8a8e44f7bc50a162
I20250626 02:04:00.855551 26185 ts_tablet_manager.cc:927] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: Initiating tablet copy from peer dac645f0cf5f4399819aea164f760471 (127.19.220.131:37981)
I20250626 02:04:00.855908 26185 tablet_copy_client.cc:323] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:37981
I20250626 02:04:00.856410 26021 tablet_copy_service.cc:140] P dac645f0cf5f4399819aea164f760471: Received BeginTabletCopySession request for tablet a7c051f83b10430185079f2255397885 from peer 6da28d4a1cd14deb8a8e44f7bc50a162 ({username='slave'} at 127.0.0.1:33010)
I20250626 02:04:00.856515 26021 tablet_copy_service.cc:161] P dac645f0cf5f4399819aea164f760471: Beginning new tablet copy session on tablet a7c051f83b10430185079f2255397885 from peer 6da28d4a1cd14deb8a8e44f7bc50a162 at {username='slave'} at 127.0.0.1:33010: session id = 6da28d4a1cd14deb8a8e44f7bc50a162-a7c051f83b10430185079f2255397885
I20250626 02:04:00.857281 26021 tablet_copy_source_session.cc:215] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:00.858042 26185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c051f83b10430185079f2255397885. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:00.859802 26185 tablet_copy_client.cc:806] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:00.859951 26185 tablet_copy_client.cc:670] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:00.860940 26185 tablet_copy_client.cc:538] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:00.862012 26185 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap starting.
I20250626 02:04:00.864917 26185 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:00.865134 26185 tablet_bootstrap.cc:492] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap complete.
I20250626 02:04:00.865285 26185 ts_tablet_manager.cc:1397] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250626 02:04:00.865653 26185 raft_consensus.cc:357] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.865764 26185 raft_consensus.cc:738] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6da28d4a1cd14deb8a8e44f7bc50a162, State: Initialized, Role: LEARNER
I20250626 02:04:00.865880 26185 consensus_queue.cc:260] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:00.866124 26185 ts_tablet_manager.cc:1428] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:00.866662 26021 tablet_copy_service.cc:342] P dac645f0cf5f4399819aea164f760471: Request end of tablet copy session 6da28d4a1cd14deb8a8e44f7bc50a162-a7c051f83b10430185079f2255397885 received from {username='slave'} at 127.0.0.1:33010
I20250626 02:04:00.866760 26021 tablet_copy_service.cc:434] P dac645f0cf5f4399819aea164f760471: ending tablet copy session 6da28d4a1cd14deb8a8e44f7bc50a162-a7c051f83b10430185079f2255397885 on tablet a7c051f83b10430185079f2255397885 with peer 6da28d4a1cd14deb8a8e44f7bc50a162
W20250626 02:04:00.978482 25975 consensus_peers.cc:487] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:01.006620 26147 raft_consensus.cc:1215] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 02:04:01.061708 25936 raft_consensus.cc:1273] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 02:04:01.062112 26184 consensus_queue.cc:1035] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:01.282927 26147 raft_consensus.cc:1215] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 02:04:01.307822 26202 consensus_queue.cc:579] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Leader has been unable to successfully communicate with peer 5b1b94cebb2948a3ad24d59990ed8072 for more than 1 seconds (1.206s)
I20250626 02:04:01.382033 26147 raft_consensus.cc:1215] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 02:04:01.401546 26197 raft_consensus.cc:1062] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5: attempting to promote NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 to VOTER
I20250626 02:04:01.401970 26197 consensus_queue.cc:237] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } }
I20250626 02:04:01.403204 26147 raft_consensus.cc:1273] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Refusing update from remote peer f275d2c9647047109d009a08b8f62cd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:01.403204 26011 raft_consensus.cc:1273] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Refusing update from remote peer f275d2c9647047109d009a08b8f62cd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:01.403599 26204 consensus_queue.cc:1035] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:01.403712 26197 consensus_queue.cc:1035] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 02:04:01.403985 25900 consensus_peers.cc:487] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:01.405229 26204 raft_consensus.cc:2953] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.405648 26147 raft_consensus.cc:2953] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.407119 26011 raft_consensus.cc:2953] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.407567 25756 catalog_manager.cc:5582] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 reported cstate change: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f275d2c9647047109d009a08b8f62cd5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:01.410645 25936 consensus_queue.cc:237] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } }
I20250626 02:04:01.411634 26011 raft_consensus.cc:1273] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Refusing update from remote peer f275d2c9647047109d009a08b8f62cd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:01.411676 26147 raft_consensus.cc:1273] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Refusing update from remote peer f275d2c9647047109d009a08b8f62cd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:01.411984 26204 consensus_queue.cc:1035] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:01.412091 26197 consensus_queue.cc:1035] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:01.413421 26196 raft_consensus.cc:2953] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.413796 26011 raft_consensus.cc:2953] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.413796 26147 raft_consensus.cc:2953] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.414855 25743 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bf07004c4a034218a0912eafd06537fb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:01.415920 25757 catalog_manager.cc:5582] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 reported cstate change: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "f275d2c9647047109d009a08b8f62cd5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 02:04:01.418860 25742 catalog_manager.cc:4671] TS 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bf07004c4a034218a0912eafd06537fb: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
W20250626 02:04:01.584095 25805 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 5b1b94cebb2948a3ad24d59990ed8072 not available for placement
I20250626 02:04:01.771813 26202 raft_consensus.cc:1062] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471: attempting to promote NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 to VOTER
I20250626 02:04:01.772238 26202 consensus_queue.cc:237] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } }
I20250626 02:04:01.773530 26147 raft_consensus.cc:1273] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:01.773545 25936 raft_consensus.cc:1273] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:01.774029 26213 consensus_queue.cc:1035] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:01.774187 26199 consensus_queue.cc:1035] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 02:04:01.774291 25975 consensus_peers.cc:487] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:01.775718 26213 raft_consensus.cc:2953] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.776154 26147 raft_consensus.cc:2953] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.777720 26213 consensus_queue.cc:579] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Leader has been unable to successfully communicate with peer 5b1b94cebb2948a3ad24d59990ed8072 for more than 1 seconds (1.459s)
I20250626 02:04:01.778196 25756 catalog_manager.cc:5582] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 reported cstate change: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.778264 25936 raft_consensus.cc:2953] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.779706 26199 raft_consensus.cc:1062] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471: attempting to promote NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 to VOTER
I20250626 02:04:01.780074 26199 consensus_queue.cc:237] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } }
I20250626 02:04:01.781258 25936 raft_consensus.cc:1273] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:01.782088 26202 consensus_queue.cc:1035] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:01.782363 26011 consensus_queue.cc:237] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } }
W20250626 02:04:01.782444 25975 consensus_peers.cc:487] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:01.783046 26147 raft_consensus.cc:1273] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 LEARNER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:01.783337 26147 raft_consensus.cc:1273] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:01.783608 26184 consensus_queue.cc:1035] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:01.783826 26184 consensus_queue.cc:1035] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:01.785051 25936 raft_consensus.cc:1273] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:01.785300 26202 raft_consensus.cc:2953] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.785531 26199 raft_consensus.cc:2953] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.786659 26202 consensus_queue.cc:1035] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:01.786739 25741 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9ac3e843d7ec4fb787aa86c0716e978d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:01.787199 26147 raft_consensus.cc:2953] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.787272 26146 raft_consensus.cc:2953] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.787416 25936 raft_consensus.cc:2953] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.787441 25935 raft_consensus.cc:2953] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.789340 25756 catalog_manager.cc:5582] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 reported cstate change: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.791141 25757 catalog_manager.cc:5582] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 reported cstate change: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:01.793593 26011 consensus_queue.cc:237] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } }
I20250626 02:04:01.794555 26146 raft_consensus.cc:1273] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:01.794713 25936 raft_consensus.cc:1273] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:01.795008 26219 consensus_queue.cc:1035] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:01.795135 26223 consensus_queue.cc:1035] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:01.796376 26223 raft_consensus.cc:2953] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.796797 25936 raft_consensus.cc:2953] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.796921 26146 raft_consensus.cc:2953] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:01.798408 25741 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a7c051f83b10430185079f2255397885 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:01.798911 25756 catalog_manager.cc:5582] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 reported cstate change: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:02.065281 26219 consensus_queue.cc:579] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Leader has been unable to successfully communicate with peer 5b1b94cebb2948a3ad24d59990ed8072 for more than 1 seconds (1.538s)
I20250626 02:04:02.067593 26011 consensus_queue.cc:237] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:02.068876 25936 raft_consensus.cc:1273] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250626 02:04:02.069128 25974 consensus_peers.cc:487] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 -> Peer 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132:34013): Couldn't send request to peer 6da28d4a1cd14deb8a8e44f7bc50a162. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c2398a33bdf417ea40325c660368946. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:02.069233 26223 consensus_queue.cc:1035] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 02:04:02.069545 25975 consensus_peers.cc:487] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:02.070851 26219 raft_consensus.cc:2953] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } } }
I20250626 02:04:02.071175 25936 raft_consensus.cc:2953] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } } }
I20250626 02:04:02.072553 25741 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6c2398a33bdf417ea40325c660368946 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 02:04:02.072957 25757 catalog_manager.cc:5582] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 reported cstate change: config changed from index -1 to 2, NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) added. New cstate: current_term: 2 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 02:04:02.444469 25742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20250626 02:04:02.445082 25742 catalog_manager.cc:4671] TS 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9ac3e843d7ec4fb787aa86c0716e978d: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
W20250626 02:04:02.584659 25805 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 5b1b94cebb2948a3ad24d59990ed8072 not available for placement
I20250626 02:04:02.657903 26228 ts_tablet_manager.cc:927] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: Initiating tablet copy from peer dac645f0cf5f4399819aea164f760471 (127.19.220.131:37981)
I20250626 02:04:02.658252 26228 tablet_copy_client.cc:323] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:37981
I20250626 02:04:02.658699 26021 tablet_copy_service.cc:140] P dac645f0cf5f4399819aea164f760471: Received BeginTabletCopySession request for tablet 6c2398a33bdf417ea40325c660368946 from peer 6da28d4a1cd14deb8a8e44f7bc50a162 ({username='slave'} at 127.0.0.1:33010)
I20250626 02:04:02.658833 26021 tablet_copy_service.cc:161] P dac645f0cf5f4399819aea164f760471: Beginning new tablet copy session on tablet 6c2398a33bdf417ea40325c660368946 from peer 6da28d4a1cd14deb8a8e44f7bc50a162 at {username='slave'} at 127.0.0.1:33010: session id = 6da28d4a1cd14deb8a8e44f7bc50a162-6c2398a33bdf417ea40325c660368946
I20250626 02:04:02.659598 26021 tablet_copy_source_session.cc:215] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:02.660449 26228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c2398a33bdf417ea40325c660368946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:02.662340 26228 tablet_copy_client.cc:806] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:02.662500 26228 tablet_copy_client.cc:670] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:02.663476 26228 tablet_copy_client.cc:538] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:02.664850 26228 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap starting.
I20250626 02:04:02.668457 26228 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:02.668702 26228 tablet_bootstrap.cc:492] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: Bootstrap complete.
I20250626 02:04:02.668892 26228 ts_tablet_manager.cc:1397] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250626 02:04:02.669306 26228 raft_consensus.cc:357] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:02.669409 26228 raft_consensus.cc:738] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6da28d4a1cd14deb8a8e44f7bc50a162, State: Initialized, Role: LEARNER
I20250626 02:04:02.669524 26228 consensus_queue.cc:260] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: true } }
I20250626 02:04:02.669883 26228 ts_tablet_manager.cc:1428] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:02.670390 26021 tablet_copy_service.cc:342] P dac645f0cf5f4399819aea164f760471: Request end of tablet copy session 6da28d4a1cd14deb8a8e44f7bc50a162-6c2398a33bdf417ea40325c660368946 received from {username='slave'} at 127.0.0.1:33010
I20250626 02:04:02.670464 26021 tablet_copy_service.cc:434] P dac645f0cf5f4399819aea164f760471: ending tablet copy session 6da28d4a1cd14deb8a8e44f7bc50a162-6c2398a33bdf417ea40325c660368946 on tablet 6c2398a33bdf417ea40325c660368946 with peer 6da28d4a1cd14deb8a8e44f7bc50a162
I20250626 02:04:03.105934 26223 consensus_queue.cc:579] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Leader has been unable to successfully communicate with peer 5b1b94cebb2948a3ad24d59990ed8072 for more than 1 seconds (1.038s)
I20250626 02:04:03.227931 26146 raft_consensus.cc:1215] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20250626 02:04:03.585237 25805 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 5b1b94cebb2948a3ad24d59990ed8072 not available for placement
W20250626 02:04:03.703023 25742 catalog_manager.cc:4671] TS 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bf07004c4a034218a0912eafd06537fb: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
I20250626 02:04:03.796389 26234 raft_consensus.cc:1062] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471: attempting to promote NON_VOTER 6da28d4a1cd14deb8a8e44f7bc50a162 to VOTER
I20250626 02:04:03.796837 26234 consensus_queue.cc:237] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } }
I20250626 02:04:03.798156 25936 raft_consensus.cc:1273] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 02:04:03.798206 26146 raft_consensus.cc:1273] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 LEARNER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 02:04:03.798581 26242 consensus_queue.cc:1035] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:03.798766 26240 consensus_queue.cc:1035] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 02:04:03.798983 25975 consensus_peers.cc:487] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 -> Peer 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): Couldn't send request to peer 5b1b94cebb2948a3ad24d59990ed8072. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:03.800477 26240 raft_consensus.cc:2953] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:03.800935 25936 raft_consensus.cc:2953] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:03.802258 26146 raft_consensus.cc:2953] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:03.802829 25755 catalog_manager.cc:5582] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 reported cstate change: config changed from index 2 to 3, 6da28d4a1cd14deb8a8e44f7bc50a162 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b1b94cebb2948a3ad24d59990ed8072" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 46201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:03.805838 26011 consensus_queue.cc:237] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } }
I20250626 02:04:03.806824 26146 raft_consensus.cc:1273] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 02:04:03.807143 26243 consensus_queue.cc:1035] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:03.807359 25936 raft_consensus.cc:1273] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Refusing update from remote peer dac645f0cf5f4399819aea164f760471: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 02:04:03.807724 26243 consensus_queue.cc:1035] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:03.808619 26242 raft_consensus.cc:2953] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:03.809042 26146 raft_consensus.cc:2953] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:03.809574 25936 raft_consensus.cc:2953] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } } }
I20250626 02:04:03.810096 25741 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6c2398a33bdf417ea40325c660368946 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:03.811151 25757 catalog_manager.cc:5582] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 reported cstate change: config changed from index 3 to 4, VOTER 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129) evicted. New cstate: current_term: 2 leader_uuid: "dac645f0cf5f4399819aea164f760471" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f275d2c9647047109d009a08b8f62cd5" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 39285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dac645f0cf5f4399819aea164f760471" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 37981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6da28d4a1cd14deb8a8e44f7bc50a162" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 34013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 02:04:05.023011 25742 catalog_manager.cc:4671] TS 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6c2398a33bdf417ea40325c660368946: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
W20250626 02:04:06.091853 25742 catalog_manager.cc:4671] TS 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6c2398a33bdf417ea40325c660368946: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
I20250626 02:04:07.094035 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:04:07.097671 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:07.097915 20338 tablet_replica.cc:331] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5: stopping tablet replica
I20250626 02:04:07.098048 20338 raft_consensus.cc:2241] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:07.098271 20338 raft_consensus.cc:2270] T bf07004c4a034218a0912eafd06537fb P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.098747 20338 tablet_replica.cc:331] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5: stopping tablet replica
I20250626 02:04:07.098855 20338 raft_consensus.cc:2241] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:07.098974 20338 raft_consensus.cc:2270] T 9ac3e843d7ec4fb787aa86c0716e978d P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.099293 20338 tablet_replica.cc:331] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5: stopping tablet replica
I20250626 02:04:07.099386 20338 raft_consensus.cc:2241] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:07.099491 20338 raft_consensus.cc:2270] T 6c2398a33bdf417ea40325c660368946 P f275d2c9647047109d009a08b8f62cd5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.099782 20338 tablet_replica.cc:331] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5: stopping tablet replica
I20250626 02:04:07.099870 20338 raft_consensus.cc:2241] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:07.099975 20338 raft_consensus.cc:2270] T a7c051f83b10430185079f2255397885 P f275d2c9647047109d009a08b8f62cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 02:04:07.101388 25900 connection.cc:441] server connection from 127.0.0.1:47356 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47356 (request call id 77) could send its response
W20250626 02:04:07.102715 25976 consensus_peers.cc:487] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 -> Peer f275d2c9647047109d009a08b8f62cd5 (127.19.220.130:39285): Couldn't send request to peer f275d2c9647047109d009a08b8f62cd5. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.130:39285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:07.113063 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:04:07.115479 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:04:07.120118 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:07.120321 20338 tablet_replica.cc:331] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471: stopping tablet replica
I20250626 02:04:07.120447 20338 raft_consensus.cc:2241] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:07.120580 20338 raft_consensus.cc:2270] T bf07004c4a034218a0912eafd06537fb P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.121001 20338 tablet_replica.cc:331] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471: stopping tablet replica
I20250626 02:04:07.121073 20338 raft_consensus.cc:2241] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:07.121223 20338 raft_consensus.cc:2270] T a7c051f83b10430185079f2255397885 P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.121541 20338 tablet_replica.cc:331] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471: stopping tablet replica
I20250626 02:04:07.121611 20338 raft_consensus.cc:2241] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:04:07.121726 20338 raft_consensus.cc:2270] T 6c2398a33bdf417ea40325c660368946 P dac645f0cf5f4399819aea164f760471 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.121989 20338 tablet_replica.cc:331] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471: stopping tablet replica
I20250626 02:04:07.122058 20338 raft_consensus.cc:2241] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:07.122175 20338 raft_consensus.cc:2270] T 9ac3e843d7ec4fb787aa86c0716e978d P dac645f0cf5f4399819aea164f760471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.135195 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:04:07.137459 20338 tablet_server.cc:178] TabletServer@127.19.220.132:0 shutting down...
I20250626 02:04:07.140684 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:07.140873 20338 tablet_replica.cc:331] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162: stopping tablet replica
I20250626 02:04:07.140985 20338 raft_consensus.cc:2241] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:07.141119 20338 raft_consensus.cc:2270] T 6c2398a33bdf417ea40325c660368946 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.141541 20338 tablet_replica.cc:331] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162: stopping tablet replica
I20250626 02:04:07.141641 20338 raft_consensus.cc:2241] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:07.141757 20338 raft_consensus.cc:2270] T a7c051f83b10430185079f2255397885 P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.142064 20338 tablet_replica.cc:331] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162: stopping tablet replica
I20250626 02:04:07.142158 20338 raft_consensus.cc:2241] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:07.142256 20338 raft_consensus.cc:2270] T bf07004c4a034218a0912eafd06537fb P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.142539 20338 tablet_replica.cc:331] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162: stopping tablet replica
I20250626 02:04:07.142642 20338 raft_consensus.cc:2241] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:07.142746 20338 raft_consensus.cc:2270] T 9ac3e843d7ec4fb787aa86c0716e978d P 6da28d4a1cd14deb8a8e44f7bc50a162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:07.155395 20338 tablet_server.cc:195] TabletServer@127.19.220.132:0 shutdown complete.
I20250626 02:04:07.157517 20338 master.cc:561] Master@127.19.220.190:42933 shutting down...
W20250626 02:04:08.180454 25742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20250626 02:04:08.181175 25742 catalog_manager.cc:4671] TS 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6c2398a33bdf417ea40325c660368946: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
W20250626 02:04:09.868855 25742 catalog_manager.cc:4671] TS 5b1b94cebb2948a3ad24d59990ed8072 (127.19.220.129:46201): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bf07004c4a034218a0912eafd06537fb: Network error: Client connection negotiation failed: client connection to 127.19.220.129:46201: connect: Connection refused (error 111)
I20250626 02:04:10.751809 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:10.752048 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:10.752107 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P c8588d89d26f457681b10c470ede261c: stopping tablet replica
I20250626 02:04:10.764952 20338 master.cc:583] Master@127.19.220.190:42933 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13222 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20250626 02:04:10.773728 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:42575
I20250626 02:04:10.774060 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:10.775874 26262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:10.775980 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:10.776010 26264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:10.775883 26261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:10.776324 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:10.776367 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:10.776382 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903450776382 us; error 0 us; skew 500 ppm
I20250626 02:04:10.777185 20338 webserver.cc:469] Webserver started at http://127.19.220.190:35241/ using document root <none> and password file <none>
I20250626 02:04:10.777303 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:10.777341 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:10.777395 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:10.777738 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "db81ca31d2ea44b1ad8641078447b44b"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:10.779126 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:04:10.779927 26269 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.780171 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:10.780234 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "db81ca31d2ea44b1ad8641078447b44b"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:10.780299 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:10.809990 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:10.810380 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:10.814728 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:42575
I20250626 02:04:10.817768 26321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:42575 every 8 connection(s)
I20250626 02:04:10.818265 26322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:10.820202 26322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b: Bootstrap starting.
I20250626 02:04:10.821004 26322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:10.821957 26322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b: No bootstrap required, opened a new log
I20250626 02:04:10.822329 26322 raft_consensus.cc:357] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db81ca31d2ea44b1ad8641078447b44b" member_type: VOTER }
I20250626 02:04:10.822443 26322 raft_consensus.cc:383] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:10.822489 26322 raft_consensus.cc:738] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db81ca31d2ea44b1ad8641078447b44b, State: Initialized, Role: FOLLOWER
I20250626 02:04:10.822641 26322 consensus_queue.cc:260] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db81ca31d2ea44b1ad8641078447b44b" member_type: VOTER }
I20250626 02:04:10.822710 26322 raft_consensus.cc:397] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:04:10.822750 26322 raft_consensus.cc:491] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:04:10.822811 26322 raft_consensus.cc:3058] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:10.823482 26322 raft_consensus.cc:513] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db81ca31d2ea44b1ad8641078447b44b" member_type: VOTER }
I20250626 02:04:10.823597 26322 leader_election.cc:304] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db81ca31d2ea44b1ad8641078447b44b; no voters:
I20250626 02:04:10.823779 26322 leader_election.cc:290] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:04:10.823861 26325 raft_consensus.cc:2802] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:10.824079 26325 raft_consensus.cc:695] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 1 LEADER]: Becoming Leader. State: Replica: db81ca31d2ea44b1ad8641078447b44b, State: Running, Role: LEADER
I20250626 02:04:10.824239 26322 sys_catalog.cc:564] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:04:10.824255 26325 consensus_queue.cc:237] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db81ca31d2ea44b1ad8641078447b44b" member_type: VOTER }
I20250626 02:04:10.824702 26326 sys_catalog.cc:455] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db81ca31d2ea44b1ad8641078447b44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db81ca31d2ea44b1ad8641078447b44b" member_type: VOTER } }
I20250626 02:04:10.824735 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [sys.catalog]: SysCatalogTable state changed. Reason: New leader db81ca31d2ea44b1ad8641078447b44b. Latest consensus state: current_term: 1 leader_uuid: "db81ca31d2ea44b1ad8641078447b44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db81ca31d2ea44b1ad8641078447b44b" member_type: VOTER } }
I20250626 02:04:10.824965 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [sys.catalog]: This master's current role is: LEADER
I20250626 02:04:10.824877 26326 sys_catalog.cc:458] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [sys.catalog]: This master's current role is: LEADER
I20250626 02:04:10.825476 26332 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:04:10.826285 26332 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:04:10.826530 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:04:10.827984 26332 catalog_manager.cc:1349] Generated new cluster ID: 814897ca448f48a39c87fa47fde09bde
I20250626 02:04:10.828033 26332 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:04:10.836730 26332 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:04:10.837261 26332 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:04:10.841989 26332 catalog_manager.cc:5955] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b: Generated new TSK 0
I20250626 02:04:10.842149 26332 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:04:10.858939 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:10.861070 26344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:10.861233 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:10.861058 26343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:10.861056 26346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:10.861554 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:10.861599 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:10.861616 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903450861615 us; error 0 us; skew 500 ppm
I20250626 02:04:10.862581 20338 webserver.cc:469] Webserver started at http://127.19.220.129:43113/ using document root <none> and password file <none>
I20250626 02:04:10.862753 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:10.862805 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:10.862866 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:10.863262 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "c2333615c8734156a49767c0087e8a42"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:10.864816 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:04:10.865794 26351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.866052 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:10.866112 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "c2333615c8734156a49767c0087e8a42"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:10.866190 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:10.876261 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:10.876688 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:10.876962 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:10.877413 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:10.877450 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.877486 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:10.877518 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.881629 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:39605
I20250626 02:04:10.881659 26414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:39605 every 8 connection(s)
I20250626 02:04:10.882473 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:10.884256 26419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:10.884294 26422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:10.884536 26420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:10.884631 20338 server_base.cc:1048] running on GCE node
I20250626 02:04:10.884783 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:10.884811 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:10.884826 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903450884826 us; error 0 us; skew 500 ppm
I20250626 02:04:10.885778 20338 webserver.cc:469] Webserver started at http://127.19.220.130:36247/ using document root <none> and password file <none>
I20250626 02:04:10.885934 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:10.885979 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:10.886041 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:10.886497 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "74cf8df2c3aa488195da158f4be08d27"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:10.887159 26415 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42575
I20250626 02:04:10.887233 26415 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:10.887398 26415 heartbeater.cc:507] Master 127.19.220.190:42575 requested a full tablet report, sending...
I20250626 02:04:10.888056 26286 ts_manager.cc:194] Registered new tserver with Master: c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605)
I20250626 02:04:10.888235 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:10.889050 26286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34172
I20250626 02:04:10.889340 26427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.889647 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:10.889735 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "74cf8df2c3aa488195da158f4be08d27"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:10.889808 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:10.902756 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:10.903168 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:10.903429 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:10.903894 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:10.903934 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.903970 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:10.904000 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.908277 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:33233
I20250626 02:04:10.908324 26490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:33233 every 8 connection(s)
I20250626 02:04:10.909075 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:10.910903 26494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:10.910910 26495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:10.910910 26497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:10.911175 20338 server_base.cc:1048] running on GCE node
I20250626 02:04:10.911406 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:10.911451 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:10.911468 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903450911467 us; error 0 us; skew 500 ppm
I20250626 02:04:10.912454 20338 webserver.cc:469] Webserver started at http://127.19.220.131:34613/ using document root <none> and password file <none>
I20250626 02:04:10.912603 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:10.912658 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:10.912755 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:10.913234 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "56533e22a80a4098912aa35714c0da66"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:10.913493 26491 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42575
I20250626 02:04:10.913594 26491 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:10.913846 26491 heartbeater.cc:507] Master 127.19.220.190:42575 requested a full tablet report, sending...
I20250626 02:04:10.914428 26286 ts_manager.cc:194] Registered new tserver with Master: 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233)
I20250626 02:04:10.915019 26286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34188
I20250626 02:04:10.915052 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250626 02:04:10.916002 26502 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.916244 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:10.916302 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "56533e22a80a4098912aa35714c0da66"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:10.916365 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:10.937475 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:10.937922 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:10.938220 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:10.938717 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:10.938758 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.938794 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:10.938831 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:10.943104 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:35229
I20250626 02:04:10.943166 26565 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:35229 every 8 connection(s)
I20250626 02:04:10.947516 26566 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42575
I20250626 02:04:10.947618 26566 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:10.947832 26566 heartbeater.cc:507] Master 127.19.220.190:42575 requested a full tablet report, sending...
I20250626 02:04:10.948446 26286 ts_manager.cc:194] Registered new tserver with Master: 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:10.948957 26286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34192
I20250626 02:04:10.949158 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005632062s
I20250626 02:04:11.889992 26415 heartbeater.cc:499] Master 127.19.220.190:42575 was elected leader, sending a full tablet report...
I20250626 02:04:11.916049 26491 heartbeater.cc:499] Master 127.19.220.190:42575 was elected leader, sending a full tablet report...
I20250626 02:04:11.949818 26566 heartbeater.cc:499] Master 127.19.220.190:42575 was elected leader, sending a full tablet report...
I20250626 02:04:11.975217 20338 test_util.cc:276] Using random seed: -95781330
I20250626 02:04:11.983580 26286 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34194:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:04:11.984179 26286 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:04:11.994405 26530 tablet_service.cc:1468] Processing CreateTablet for tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:11.994557 26527 tablet_service.cc:1468] Processing CreateTablet for tablet c86b7798500d4ebea91b6d389c68ed1b (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:11.994856 26529 tablet_service.cc:1468] Processing CreateTablet for tablet 22af9a7f4a564cb7b19bf75222040896 (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:11.995613 26455 tablet_service.cc:1468] Processing CreateTablet for tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:11.995877 26455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:11.996011 26379 tablet_service.cc:1468] Processing CreateTablet for tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:11.996068 26454 tablet_service.cc:1468] Processing CreateTablet for tablet 22af9a7f4a564cb7b19bf75222040896 (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:11.996268 26379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:11.995818 26452 tablet_service.cc:1468] Processing CreateTablet for tablet c86b7798500d4ebea91b6d389c68ed1b (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:11.995015 26528 tablet_service.cc:1468] Processing CreateTablet for tablet 292cdbfda66145d98d69d39f4811ed37 (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:11.996272 26454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22af9a7f4a564cb7b19bf75222040896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:11.994971 26530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:11.998590 26452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86b7798500d4ebea91b6d389c68ed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:11.998816 26528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292cdbfda66145d98d69d39f4811ed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:11.999006 26377 tablet_service.cc:1468] Processing CreateTablet for tablet 292cdbfda66145d98d69d39f4811ed37 (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:11.995818 26453 tablet_service.cc:1468] Processing CreateTablet for tablet 292cdbfda66145d98d69d39f4811ed37 (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:11.999243 26377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292cdbfda66145d98d69d39f4811ed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:11.999348 26453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292cdbfda66145d98d69d39f4811ed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:12.000871 26588 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Bootstrap starting.
I20250626 02:04:12.001092 26529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22af9a7f4a564cb7b19bf75222040896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:12.001370 26376 tablet_service.cc:1468] Processing CreateTablet for tablet c86b7798500d4ebea91b6d389c68ed1b (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:12.001621 26376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86b7798500d4ebea91b6d389c68ed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:12.001827 26588 tablet_bootstrap.cc:654] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.003064 26588 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: No bootstrap required, opened a new log
I20250626 02:04:12.003122 26527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86b7798500d4ebea91b6d389c68ed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:12.003144 26588 ts_tablet_manager.cc:1397] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:12.003497 26588 raft_consensus.cc:357] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:12.003600 26588 raft_consensus.cc:383] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.003633 26588 raft_consensus.cc:738] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.003777 26588 consensus_queue.cc:260] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:11.998822 26378 tablet_service.cc:1468] Processing CreateTablet for tablet 22af9a7f4a564cb7b19bf75222040896 (DEFAULT_TABLE table=test-workload [id=b54c597be49540ae9b980507c1197eab]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:12.004159 26378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22af9a7f4a564cb7b19bf75222040896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:12.004771 26587 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Bootstrap starting.
I20250626 02:04:12.005259 26586 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Bootstrap starting.
I20250626 02:04:12.005801 26587 tablet_bootstrap.cc:654] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.004155 26588 ts_tablet_manager.cc:1428] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:12.006162 26588 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Bootstrap starting.
I20250626 02:04:12.006217 26586 tablet_bootstrap.cc:654] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.007184 26588 tablet_bootstrap.cc:654] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.007822 26586 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: No bootstrap required, opened a new log
I20250626 02:04:12.007925 26586 ts_tablet_manager.cc:1397] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:12.008291 26588 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: No bootstrap required, opened a new log
I20250626 02:04:12.008424 26588 ts_tablet_manager.cc:1397] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:12.008414 26586 raft_consensus.cc:357] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:12.008523 26586 raft_consensus.cc:383] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.008565 26586 raft_consensus.cc:738] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf8df2c3aa488195da158f4be08d27, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.008617 26587 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: No bootstrap required, opened a new log
I20250626 02:04:12.008695 26587 ts_tablet_manager.cc:1397] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:04:12.008721 26586 consensus_queue.cc:260] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:12.008886 26588 raft_consensus.cc:357] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.008975 26588 raft_consensus.cc:383] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.009011 26588 raft_consensus.cc:738] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.009056 26586 ts_tablet_manager.cc:1428] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:12.009104 26587 raft_consensus.cc:357] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:12.009123 26588 consensus_queue.cc:260] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.009181 26587 raft_consensus.cc:383] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.009212 26587 raft_consensus.cc:738] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.009230 26586 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Bootstrap starting.
I20250626 02:04:12.009318 26587 consensus_queue.cc:260] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:12.009382 26588 ts_tablet_manager.cc:1428] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:12.009557 26588 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Bootstrap starting.
I20250626 02:04:12.010510 26588 tablet_bootstrap.cc:654] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.010841 26586 tablet_bootstrap.cc:654] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.011678 26588 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: No bootstrap required, opened a new log
I20250626 02:04:12.011757 26588 ts_tablet_manager.cc:1397] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:04:12.012197 26588 raft_consensus.cc:357] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.012290 26588 raft_consensus.cc:383] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.012318 26588 raft_consensus.cc:738] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.012425 26588 consensus_queue.cc:260] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.012681 26588 ts_tablet_manager.cc:1428] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:12.012847 26588 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Bootstrap starting.
I20250626 02:04:12.012992 26587 ts_tablet_manager.cc:1428] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250626 02:04:12.013203 26587 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Bootstrap starting.
I20250626 02:04:12.013845 26588 tablet_bootstrap.cc:654] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.014796 26587 tablet_bootstrap.cc:654] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.015772 26588 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: No bootstrap required, opened a new log
I20250626 02:04:12.015857 26588 ts_tablet_manager.cc:1397] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:12.016002 26586 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: No bootstrap required, opened a new log
I20250626 02:04:12.016091 26586 ts_tablet_manager.cc:1397] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250626 02:04:12.016355 26588 raft_consensus.cc:357] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.016469 26588 raft_consensus.cc:383] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.016512 26588 raft_consensus.cc:738] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.016523 26586 raft_consensus.cc:357] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.016638 26588 consensus_queue.cc:260] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.016932 26588 ts_tablet_manager.cc:1428] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:12.016640 26586 raft_consensus.cc:383] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.017262 26586 raft_consensus.cc:738] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf8df2c3aa488195da158f4be08d27, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.017386 26586 consensus_queue.cc:260] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.017637 26586 ts_tablet_manager.cc:1428] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:04:12.017818 26586 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Bootstrap starting.
I20250626 02:04:12.018828 26586 tablet_bootstrap.cc:654] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.020202 26586 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: No bootstrap required, opened a new log
I20250626 02:04:12.020295 26586 ts_tablet_manager.cc:1397] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:12.020349 26587 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: No bootstrap required, opened a new log
I20250626 02:04:12.020444 26587 ts_tablet_manager.cc:1397] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250626 02:04:12.020699 26586 raft_consensus.cc:357] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.020789 26586 raft_consensus.cc:383] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.020823 26586 raft_consensus.cc:738] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf8df2c3aa488195da158f4be08d27, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.020833 26587 raft_consensus.cc:357] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.020951 26586 consensus_queue.cc:260] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.020956 26587 raft_consensus.cc:383] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.021077 26587 raft_consensus.cc:738] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.021188 26587 consensus_queue.cc:260] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.021237 26586 ts_tablet_manager.cc:1428] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:12.021389 26586 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Bootstrap starting.
I20250626 02:04:12.021431 26587 ts_tablet_manager.cc:1428] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:12.021575 26587 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Bootstrap starting.
I20250626 02:04:12.022368 26586 tablet_bootstrap.cc:654] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.022387 26587 tablet_bootstrap.cc:654] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.023525 26587 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: No bootstrap required, opened a new log
I20250626 02:04:12.023617 26587 ts_tablet_manager.cc:1397] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:04:12.023717 26586 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: No bootstrap required, opened a new log
I20250626 02:04:12.023792 26586 ts_tablet_manager.cc:1397] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:04:12.024046 26587 raft_consensus.cc:357] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.024191 26587 raft_consensus.cc:383] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.024238 26587 raft_consensus.cc:738] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.024353 26587 consensus_queue.cc:260] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.024786 26587 ts_tablet_manager.cc:1428] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:12.024739 26586 raft_consensus.cc:357] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.024935 26586 raft_consensus.cc:383] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.025012 26587 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Bootstrap starting.
I20250626 02:04:12.025009 26586 raft_consensus.cc:738] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf8df2c3aa488195da158f4be08d27, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.025168 26586 consensus_queue.cc:260] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.025543 26586 ts_tablet_manager.cc:1428] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:04:12.026110 26587 tablet_bootstrap.cc:654] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:12.027199 26587 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: No bootstrap required, opened a new log
I20250626 02:04:12.027298 26587 ts_tablet_manager.cc:1397] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:04:12.027765 26587 raft_consensus.cc:357] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.027859 26587 raft_consensus.cc:383] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:12.027890 26587 raft_consensus.cc:738] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Initialized, Role: FOLLOWER
I20250626 02:04:12.027976 26587 consensus_queue.cc:260] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.028201 26587 ts_tablet_manager.cc:1428] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:12.042411 26597 raft_consensus.cc:491] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:12.042526 26597 raft_consensus.cc:513] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.043006 26597 leader_election.cc:290] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:12.047299 26540 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66" is_pre_election: true
I20250626 02:04:12.047325 26465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:12.047430 26465 raft_consensus.cc:2466] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 0.
I20250626 02:04:12.047434 26540 raft_consensus.cc:2466] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 0.
I20250626 02:04:12.047780 26352 leader_election.cc:304] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters:
I20250626 02:04:12.047945 26597 raft_consensus.cc:2802] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:12.048010 26597 raft_consensus.cc:491] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:12.048053 26597 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.048920 26597 raft_consensus.cc:513] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.049242 26597 leader_election.cc:290] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 1 election: Requested vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:12.049515 26465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:12.049612 26540 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66"
I20250626 02:04:12.049649 26465 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.049696 26540 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.050642 26465 raft_consensus.cc:2466] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 1.
I20250626 02:04:12.050642 26540 raft_consensus.cc:2466] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 1.
I20250626 02:04:12.051014 26355 leader_election.cc:304] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, c2333615c8734156a49767c0087e8a42; no voters:
I20250626 02:04:12.051187 26597 raft_consensus.cc:2802] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:12.051376 26597 raft_consensus.cc:695] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 LEADER]: Becoming Leader. State: Replica: c2333615c8734156a49767c0087e8a42, State: Running, Role: LEADER
I20250626 02:04:12.051535 26597 consensus_queue.cc:237] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.053083 26285 catalog_manager.cc:5582] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2333615c8734156a49767c0087e8a42 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "c2333615c8734156a49767c0087e8a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: UNKNOWN } } }
W20250626 02:04:12.133253 26416 tablet.cc:2378] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 02:04:12.140668 26597 raft_consensus.cc:491] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:12.140822 26597 raft_consensus.cc:513] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.141274 26597 leader_election.cc:290] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:12.141652 26465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22af9a7f4a564cb7b19bf75222040896" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:12.141680 26540 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22af9a7f4a564cb7b19bf75222040896" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66" is_pre_election: true
I20250626 02:04:12.141790 26465 raft_consensus.cc:2466] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 0.
I20250626 02:04:12.141824 26540 raft_consensus.cc:2466] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 0.
I20250626 02:04:12.142040 26352 leader_election.cc:304] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters:
I20250626 02:04:12.142206 26597 raft_consensus.cc:2802] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:12.142262 26597 raft_consensus.cc:491] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:12.142292 26597 raft_consensus.cc:3058] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.143123 26597 raft_consensus.cc:513] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.143428 26597 leader_election.cc:290] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 1 election: Requested vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:12.143728 26465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22af9a7f4a564cb7b19bf75222040896" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:12.143712 26540 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22af9a7f4a564cb7b19bf75222040896" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66"
I20250626 02:04:12.143842 26465 raft_consensus.cc:3058] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.143869 26540 raft_consensus.cc:3058] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.144697 26540 raft_consensus.cc:2466] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 1.
I20250626 02:04:12.144716 26465 raft_consensus.cc:2466] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 1.
I20250626 02:04:12.144985 26355 leader_election.cc:304] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, c2333615c8734156a49767c0087e8a42; no voters:
I20250626 02:04:12.145176 26597 raft_consensus.cc:2802] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:12.145259 26597 raft_consensus.cc:695] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 LEADER]: Becoming Leader. State: Replica: c2333615c8734156a49767c0087e8a42, State: Running, Role: LEADER
I20250626 02:04:12.145393 26597 consensus_queue.cc:237] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.146821 26285 catalog_manager.cc:5582] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2333615c8734156a49767c0087e8a42 (127.19.220.129). New cstate: current_term: 1 leader_uuid: "c2333615c8734156a49767c0087e8a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:12.275887 26590 raft_consensus.cc:491] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:12.276055 26590 raft_consensus.cc:513] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.276672 26590 leader_election.cc:290] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605), 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233)
I20250626 02:04:12.280042 26465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86b7798500d4ebea91b6d389c68ed1b" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:12.280071 26389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86b7798500d4ebea91b6d389c68ed1b" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2333615c8734156a49767c0087e8a42" is_pre_election: true
I20250626 02:04:12.280215 26465 raft_consensus.cc:2466] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 0.
I20250626 02:04:12.280236 26389 raft_consensus.cc:2466] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 0.
I20250626 02:04:12.280524 26503 leader_election.cc:304] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27; no voters:
I20250626 02:04:12.280687 26590 raft_consensus.cc:2802] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:12.280747 26590 raft_consensus.cc:491] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:12.280788 26590 raft_consensus.cc:3058] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.281666 26590 raft_consensus.cc:513] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.281973 26590 leader_election.cc:290] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 1 election: Requested vote from peers c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605), 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233)
I20250626 02:04:12.282274 26465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86b7798500d4ebea91b6d389c68ed1b" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:12.282250 26389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86b7798500d4ebea91b6d389c68ed1b" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2333615c8734156a49767c0087e8a42"
I20250626 02:04:12.282372 26389 raft_consensus.cc:3058] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.282371 26465 raft_consensus.cc:3058] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.283324 26389 raft_consensus.cc:2466] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 1.
I20250626 02:04:12.283324 26465 raft_consensus.cc:2466] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 1.
I20250626 02:04:12.283638 26503 leader_election.cc:304] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27; no voters:
I20250626 02:04:12.283778 26590 raft_consensus.cc:2802] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:12.283972 26590 raft_consensus.cc:695] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 LEADER]: Becoming Leader. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Running, Role: LEADER
I20250626 02:04:12.284111 26590 consensus_queue.cc:237] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:12.285477 26285 catalog_manager.cc:5582] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56533e22a80a4098912aa35714c0da66 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "56533e22a80a4098912aa35714c0da66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:12.293499 26590 raft_consensus.cc:491] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:12.293587 26590 raft_consensus.cc:513] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:12.293910 26590 leader_election.cc:290] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605)
I20250626 02:04:12.294188 26465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:12.294195 26389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2333615c8734156a49767c0087e8a42" is_pre_election: true
I20250626 02:04:12.294286 26465 raft_consensus.cc:2466] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 0.
I20250626 02:04:12.294322 26389 raft_consensus.cc:2466] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 0.
I20250626 02:04:12.294533 26503 leader_election.cc:304] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27; no voters:
I20250626 02:04:12.294674 26590 raft_consensus.cc:2802] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:12.294745 26590 raft_consensus.cc:491] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:12.294773 26590 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.295569 26590 raft_consensus.cc:513] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:12.295889 26590 leader_election.cc:290] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 1 election: Requested vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605)
I20250626 02:04:12.296165 26465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:12.296206 26389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2333615c8734156a49767c0087e8a42"
I20250626 02:04:12.296257 26465 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.296319 26389 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:12.296993 26465 raft_consensus.cc:2466] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 1.
I20250626 02:04:12.297084 26389 raft_consensus.cc:2466] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 1.
I20250626 02:04:12.297288 26503 leader_election.cc:304] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27; no voters:
I20250626 02:04:12.297603 26590 raft_consensus.cc:2802] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:12.297695 26590 raft_consensus.cc:695] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 LEADER]: Becoming Leader. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Running, Role: LEADER
I20250626 02:04:12.297829 26590 consensus_queue.cc:237] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:12.299208 26285 catalog_manager.cc:5582] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56533e22a80a4098912aa35714c0da66 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "56533e22a80a4098912aa35714c0da66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:12.351653 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:04:12.355523 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:12.355794 20338 tablet_replica.cc:331] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: stopping tablet replica
I20250626 02:04:12.355919 20338 raft_consensus.cc:2241] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:12.356065 20338 pending_rounds.cc:70] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Trying to abort 1 pending ops.
I20250626 02:04:12.356103 20338 pending_rounds.cc:77] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171700539988844544 op_type: NO_OP noop_request { }
I20250626 02:04:12.356199 20338 raft_consensus.cc:2887] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:04:12.356249 20338 raft_consensus.cc:2270] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.356835 20338 tablet_replica.cc:331] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: stopping tablet replica
I20250626 02:04:12.356917 20338 raft_consensus.cc:2241] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:12.356993 20338 raft_consensus.cc:2270] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.357312 20338 tablet_replica.cc:331] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: stopping tablet replica
I20250626 02:04:12.357380 20338 raft_consensus.cc:2241] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:12.357465 20338 raft_consensus.cc:2270] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.357776 20338 tablet_replica.cc:331] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: stopping tablet replica
I20250626 02:04:12.357842 20338 raft_consensus.cc:2241] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:12.357936 20338 pending_rounds.cc:70] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Trying to abort 1 pending ops.
I20250626 02:04:12.357959 20338 pending_rounds.cc:77] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171700539604430848 op_type: NO_OP noop_request { }
I20250626 02:04:12.358009 20338 raft_consensus.cc:2887] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:04:12.358048 20338 raft_consensus.cc:2270] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.371186 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:04:12.373471 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:04:12.377022 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:12.377259 20338 tablet_replica.cc:331] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: stopping tablet replica
I20250626 02:04:12.377378 20338 raft_consensus.cc:2241] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:12.377481 20338 raft_consensus.cc:2270] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.378008 20338 tablet_replica.cc:331] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: stopping tablet replica
I20250626 02:04:12.378091 20338 raft_consensus.cc:2241] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:12.378170 20338 raft_consensus.cc:2270] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.378486 20338 tablet_replica.cc:331] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: stopping tablet replica
I20250626 02:04:12.378556 20338 raft_consensus.cc:2241] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:12.378625 20338 raft_consensus.cc:2270] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.378937 20338 tablet_replica.cc:331] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: stopping tablet replica
I20250626 02:04:12.379002 20338 raft_consensus.cc:2241] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:12.379060 20338 raft_consensus.cc:2270] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.392089 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:04:12.394371 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:04:12.398569 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:12.398859 20338 tablet_replica.cc:331] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: stopping tablet replica
I20250626 02:04:12.398978 20338 raft_consensus.cc:2241] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:12.399128 20338 pending_rounds.cc:70] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Trying to abort 1 pending ops.
I20250626 02:04:12.399155 20338 pending_rounds.cc:77] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171700540557148160 op_type: NO_OP noop_request { }
I20250626 02:04:12.399219 20338 raft_consensus.cc:2887] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:04:12.399263 20338 raft_consensus.cc:2270] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.399832 20338 tablet_replica.cc:331] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: stopping tablet replica
I20250626 02:04:12.399919 20338 raft_consensus.cc:2241] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:12.399995 20338 raft_consensus.cc:2270] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.400400 20338 tablet_replica.cc:331] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: stopping tablet replica
I20250626 02:04:12.400483 20338 raft_consensus.cc:2241] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:12.400584 20338 pending_rounds.cc:70] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Trying to abort 1 pending ops.
I20250626 02:04:12.400609 20338 pending_rounds.cc:77] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171700540613283840 op_type: NO_OP noop_request { }
I20250626 02:04:12.400658 20338 raft_consensus.cc:2887] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:04:12.400697 20338 raft_consensus.cc:2270] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.401146 20338 tablet_replica.cc:331] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: stopping tablet replica
I20250626 02:04:12.401213 20338 raft_consensus.cc:2241] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:12.401280 20338 raft_consensus.cc:2270] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:12.414156 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:04:12.416666 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:12.418210 26614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:12.418277 26617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:12.418305 26615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:12.418390 20338 server_base.cc:1048] running on GCE node
I20250626 02:04:12.418628 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:12.418678 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:12.418726 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903452418726 us; error 0 us; skew 500 ppm
I20250626 02:04:12.419663 20338 webserver.cc:469] Webserver started at http://127.19.220.132:41051/ using document root <none> and password file <none>
I20250626 02:04:12.419831 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:12.419894 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:12.419982 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:12.420429 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-3-root/instance:
uuid: "933876bfc7a4435ca9dd1a4feab14528"
format_stamp: "Formatted at 2025-06-26 02:04:12 on dist-test-slave-h18f"
I20250626 02:04:12.421850 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:12.422665 26622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:12.422868 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:12.422941 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-3-root
uuid: "933876bfc7a4435ca9dd1a4feab14528"
format_stamp: "Formatted at 2025-06-26 02:04:12 on dist-test-slave-h18f"
I20250626 02:04:12.423022 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:12.433514 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:12.433861 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:12.434157 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:12.434607 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:12.434653 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:12.434705 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:12.434742 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:12.438445 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.132:46095
I20250626 02:04:12.440594 26685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.132:46095 every 8 connection(s)
I20250626 02:04:12.444661 26686 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42575
I20250626 02:04:12.444775 26686 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:12.444979 26686 heartbeater.cc:507] Master 127.19.220.190:42575 requested a full tablet report, sending...
I20250626 02:04:12.445569 26285 ts_manager.cc:194] Registered new tserver with Master: 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132:46095)
I20250626 02:04:12.446081 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34202
W20250626 02:04:12.828336 26334 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:12.829872 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:12.829950 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:12.830951 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:12.831017 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:12.832108 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:12.832207 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
I20250626 02:04:13.446974 26686 heartbeater.cc:499] Master 127.19.220.190:42575 was elected leader, sending a full tablet report...
W20250626 02:04:13.834954 26334 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:13.835765 26273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250626 02:04:13.836561 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:13.836614 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:13.837678 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:13.837738 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:13.838564 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:13.838620 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:14.840996 26334 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:14.842361 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:14.842419 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:14.843494 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:14.843540 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:14.844491 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:14.844553 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:15.846983 26334 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:15.848377 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:15.848428 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:15.849382 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:15.849426 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:15.850342 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:15.850380 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:16.852779 26334 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:16.854192 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:16.854254 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.131:35229: connect: Connection refused (error 111)
W20250626 02:04:16.855335 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:16.855382 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:16.856410 26334 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:16.856455 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.19.220.129:39605: connect: Connection refused (error 111)
W20250626 02:04:17.857034 26334 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 56533e22a80a4098912aa35714c0da66 not available for placement
I20250626 02:04:17.953398 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:17.955366 26707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:17.955382 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:17.955442 26705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:17.955587 26704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:17.955848 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:17.955914 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:17.955932 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903457955932 us; error 0 us; skew 500 ppm
I20250626 02:04:17.959985 20338 webserver.cc:469] Webserver started at http://127.19.220.129:43113/ using document root <none> and password file <none>
I20250626 02:04:17.960196 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:17.960237 20338 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250626 02:04:17.961252 20338 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:17.962316 26712 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:17.962568 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:17.962628 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "c2333615c8734156a49767c0087e8a42"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:17.962694 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:17.974258 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:17.974622 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:17.974850 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:17.975534 26719 ts_tablet_manager.cc:536] Loading tablet metadata (0/4 complete)
I20250626 02:04:17.978561 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250626 02:04:17.978621 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250626 02:04:17.978662 20338 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20250626 02:04:17.979494 26719 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Bootstrap starting.
I20250626 02:04:17.980748 20338 ts_tablet_manager.cc:610] Registered 4 tablets
I20250626 02:04:17.980796 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:17.981925 26719 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:17.982209 26719 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Bootstrap complete.
I20250626 02:04:17.982422 26719 ts_tablet_manager.cc:1397] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:17.982964 26719 raft_consensus.cc:357] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:17.983100 26719 raft_consensus.cc:738] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Initialized, Role: FOLLOWER
I20250626 02:04:17.983304 26719 consensus_queue.cc:260] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:17.983719 26719 ts_tablet_manager.cc:1428] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:17.983919 26719 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Bootstrap starting.
I20250626 02:04:17.986179 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:39605
I20250626 02:04:17.986207 26779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:39605 every 8 connection(s)
I20250626 02:04:17.986856 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:04:17.986970 26719 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 02:04:17.987417 26719 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Bootstrap complete.
I20250626 02:04:17.987622 26719 ts_tablet_manager.cc:1397] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250626 02:04:17.988058 26719 raft_consensus.cc:357] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:17.988266 26719 raft_consensus.cc:738] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Initialized, Role: FOLLOWER
I20250626 02:04:17.988468 26719 consensus_queue.cc:260] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:17.988883 26719 ts_tablet_manager.cc:1428] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250626 02:04:17.989090 26787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:17.989187 26785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:17.989144 26719 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Bootstrap starting.
W20250626 02:04:17.989269 26784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:17.989374 20338 server_base.cc:1048] running on GCE node
I20250626 02:04:17.990962 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:17.990998 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:17.991019 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903457991019 us; error 0 us; skew 500 ppm
I20250626 02:04:17.991338 26719 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:17.991573 26719 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Bootstrap complete.
I20250626 02:04:17.991767 26719 ts_tablet_manager.cc:1397] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250626 02:04:17.992244 26719 raft_consensus.cc:357] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:17.992365 20338 webserver.cc:469] Webserver started at http://127.19.220.130:36247/ using document root <none> and password file <none>
I20250626 02:04:17.992432 26719 raft_consensus.cc:738] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Initialized, Role: FOLLOWER
I20250626 02:04:17.992543 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:17.992586 20338 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250626 02:04:17.992602 26719 consensus_queue.cc:260] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:17.992866 26719 ts_tablet_manager.cc:1428] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:17.993008 26719 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Bootstrap starting.
I20250626 02:04:17.993108 26780 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42575
I20250626 02:04:17.993191 26780 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:17.993391 26780 heartbeater.cc:507] Master 127.19.220.190:42575 requested a full tablet report, sending...
I20250626 02:04:17.994050 20338 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:17.994333 26285 ts_manager.cc:194] Re-registered known tserver with Master: c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605)
I20250626 02:04:17.995467 26719 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 02:04:17.995584 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53152
I20250626 02:04:17.995754 26719 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Bootstrap complete.
I20250626 02:04:17.995927 26719 ts_tablet_manager.cc:1397] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:17.996161 26794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:17.996322 26719 raft_consensus.cc:357] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:17.996466 20338 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250626 02:04:17.996496 26719 raft_consensus.cc:738] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Initialized, Role: FOLLOWER
I20250626 02:04:17.996476 26780 heartbeater.cc:499] Master 127.19.220.190:42575 was elected leader, sending a full tablet report...
I20250626 02:04:17.996543 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "74cf8df2c3aa488195da158f4be08d27"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:17.996637 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:17.996665 26719 consensus_queue.cc:260] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:17.996987 26719 ts_tablet_manager.cc:1428] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:18.043743 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:18.044100 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:18.044415 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:18.047979 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250626 02:04:18.048041 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250626 02:04:18.048877 26801 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Bootstrap starting.
I20250626 02:04:18.050036 20338 ts_tablet_manager.cc:610] Registered 4 tablets
I20250626 02:04:18.050081 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250626 02:04:18.051348 26801 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:18.051646 26801 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Bootstrap complete.
I20250626 02:04:18.051843 26801 ts_tablet_manager.cc:1397] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:18.052395 26801 raft_consensus.cc:357] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:18.052568 26801 raft_consensus.cc:738] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf8df2c3aa488195da158f4be08d27, State: Initialized, Role: FOLLOWER
I20250626 02:04:18.052791 26801 consensus_queue.cc:260] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:18.053190 26801 ts_tablet_manager.cc:1428] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:18.053378 26801 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Bootstrap starting.
I20250626 02:04:18.055644 26860 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:33233 every 8 connection(s)
I20250626 02:04:18.055629 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:33233
I20250626 02:04:18.056306 26801 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:18.056391 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:04:18.056571 26801 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Bootstrap complete.
I20250626 02:04:18.056747 26801 ts_tablet_manager.cc:1397] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250626 02:04:18.057189 26801 raft_consensus.cc:357] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.057284 26801 raft_consensus.cc:738] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf8df2c3aa488195da158f4be08d27, State: Initialized, Role: FOLLOWER
I20250626 02:04:18.057408 26801 consensus_queue.cc:260] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.057696 26801 ts_tablet_manager.cc:1428] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:18.057847 26801 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Bootstrap starting.
W20250626 02:04:18.058610 26866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:18.058758 26868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:18.058974 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:18.059026 26865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:18.059334 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:18.059377 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:18.059415 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903458059414 us; error 0 us; skew 500 ppm
I20250626 02:04:18.060483 20338 webserver.cc:469] Webserver started at http://127.19.220.131:34613/ using document root <none> and password file <none>
I20250626 02:04:18.060637 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:18.060679 20338 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250626 02:04:18.061333 26862 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42575
I20250626 02:04:18.061439 26862 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:18.061514 26801 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:18.061679 26862 heartbeater.cc:507] Master 127.19.220.190:42575 requested a full tablet report, sending...
I20250626 02:04:18.061750 26801 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Bootstrap complete.
I20250626 02:04:18.061902 26801 ts_tablet_manager.cc:1397] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:04:18.062335 26801 raft_consensus.cc:357] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.062458 26801 raft_consensus.cc:738] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf8df2c3aa488195da158f4be08d27, State: Initialized, Role: FOLLOWER
I20250626 02:04:18.062536 20338 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20250626 02:04:18.062565 26285 ts_manager.cc:194] Re-registered known tserver with Master: 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233)
I20250626 02:04:18.062646 26801 consensus_queue.cc:260] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.062940 26801 ts_tablet_manager.cc:1428] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:18.063087 26801 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Bootstrap starting.
I20250626 02:04:18.063949 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53156
I20250626 02:04:18.064007 26874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:18.064741 26862 heartbeater.cc:499] Master 127.19.220.190:42575 was elected leader, sending a full tablet report...
I20250626 02:04:18.065389 20338 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250626 02:04:18.065459 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "56533e22a80a4098912aa35714c0da66"
format_stamp: "Formatted at 2025-06-26 02:04:10 on dist-test-slave-h18f"
I20250626 02:04:18.065528 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:18.065622 26801 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:18.065840 26801 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Bootstrap complete.
I20250626 02:04:18.066009 26801 ts_tablet_manager.cc:1397] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:04:18.066494 26801 raft_consensus.cc:357] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.066597 26801 raft_consensus.cc:738] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf8df2c3aa488195da158f4be08d27, State: Initialized, Role: FOLLOWER
I20250626 02:04:18.066708 26801 consensus_queue.cc:260] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.066922 26801 ts_tablet_manager.cc:1428] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:18.074716 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:18.075103 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:18.075335 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:18.078774 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250626 02:04:18.078830 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250626 02:04:18.079555 26882 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Bootstrap starting.
I20250626 02:04:18.080746 20338 ts_tablet_manager.cc:610] Registered 4 tablets
I20250626 02:04:18.080790 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250626 02:04:18.082471 26882 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 02:04:18.082824 26882 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Bootstrap complete.
I20250626 02:04:18.083050 26882 ts_tablet_manager.cc:1397] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250626 02:04:18.083511 26882 raft_consensus.cc:357] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:18.083694 26882 raft_consensus.cc:738] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Initialized, Role: FOLLOWER
I20250626 02:04:18.083854 26882 consensus_queue.cc:260] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:18.084235 26882 ts_tablet_manager.cc:1428] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:18.084388 26882 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Bootstrap starting.
I20250626 02:04:18.085829 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:35229
I20250626 02:04:18.085870 26941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:35229 every 8 connection(s)
I20250626 02:04:18.086827 26882 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:18.087036 26882 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Bootstrap complete.
I20250626 02:04:18.087159 26882 ts_tablet_manager.cc:1397] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:18.087497 26882 raft_consensus.cc:357] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.087584 26882 raft_consensus.cc:738] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Initialized, Role: FOLLOWER
I20250626 02:04:18.087693 26882 consensus_queue.cc:260] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.087939 26882 ts_tablet_manager.cc:1428] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:18.088049 26882 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Bootstrap starting.
I20250626 02:04:18.090276 26942 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42575
I20250626 02:04:18.090387 26942 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:18.090629 26942 heartbeater.cc:507] Master 127.19.220.190:42575 requested a full tablet report, sending...
I20250626 02:04:18.090827 26882 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 02:04:18.091127 26882 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Bootstrap complete.
I20250626 02:04:18.091290 26882 ts_tablet_manager.cc:1397] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:18.091583 26285 ts_manager.cc:194] Re-registered known tserver with Master: 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:18.091670 26882 raft_consensus.cc:357] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.091813 26882 raft_consensus.cc:738] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Initialized, Role: FOLLOWER
I20250626 02:04:18.091954 26882 consensus_queue.cc:260] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.092233 26882 ts_tablet_manager.cc:1428] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:18.092381 26882 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Bootstrap starting.
I20250626 02:04:18.092835 26285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53170
I20250626 02:04:18.093638 26942 heartbeater.cc:499] Master 127.19.220.190:42575 was elected leader, sending a full tablet report...
I20250626 02:04:18.094722 26882 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:18.094929 26882 tablet_bootstrap.cc:492] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Bootstrap complete.
I20250626 02:04:18.095078 26882 ts_tablet_manager.cc:1397] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:18.095520 26882 raft_consensus.cc:357] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.095623 26882 raft_consensus.cc:738] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Initialized, Role: FOLLOWER
I20250626 02:04:18.095770 26882 consensus_queue.cc:260] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:18.095990 26882 ts_tablet_manager.cc:1428] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250626 02:04:18.863142 26334 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Replica c2333615c8734156a49767c0087e8a42 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } }
W20250626 02:04:18.864264 26334 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 22af9a7f4a564cb7b19bf75222040896, TS 56533e22a80a4098912aa35714c0da66 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
W20250626 02:04:18.864331 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 22af9a7f4a564cb7b19bf75222040896, TS 56533e22a80a4098912aa35714c0da66 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
W20250626 02:04:18.868193 26334 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c86b7798500d4ebea91b6d389c68ed1b, TS c2333615c8734156a49767c0087e8a42 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
W20250626 02:04:18.868261 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet c86b7798500d4ebea91b6d389c68ed1b, TS c2333615c8734156a49767c0087e8a42 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
W20250626 02:04:18.869060 26334 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 292cdbfda66145d98d69d39f4811ed37, TS 74cf8df2c3aa488195da158f4be08d27 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
W20250626 02:04:18.869117 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 292cdbfda66145d98d69d39f4811ed37, TS 74cf8df2c3aa488195da158f4be08d27 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.117156 26951 raft_consensus.cc:491] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:19.117287 26951 raft_consensus.cc:513] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:19.117821 26951 leader_election.cc:290] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229), 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233)
I20250626 02:04:19.121161 26914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66" is_pre_election: true
I20250626 02:04:19.121326 26914 raft_consensus.cc:2408] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2333615c8734156a49767c0087e8a42 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250626 02:04:19.121371 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:19.121562 26835 raft_consensus.cc:2466] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 1.
I20250626 02:04:19.121831 26713 leader_election.cc:304] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters: 56533e22a80a4098912aa35714c0da66
I20250626 02:04:19.122004 26951 raft_consensus.cc:2802] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 02:04:19.122064 26951 raft_consensus.cc:491] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:19.122108 26951 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.122977 26951 raft_consensus.cc:513] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:19.123287 26951 leader_election.cc:290] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 election: Requested vote from peers 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229), 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233)
I20250626 02:04:19.123556 26914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66"
I20250626 02:04:19.123571 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:19.123690 26914 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.123772 26835 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.124886 26835 raft_consensus.cc:2466] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 2.
I20250626 02:04:19.124935 26914 raft_consensus.cc:2408] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c2333615c8734156a49767c0087e8a42 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250626 02:04:19.125149 26713 leader_election.cc:304] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters:
I20250626 02:04:19.125367 26951 raft_consensus.cc:2802] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:04:19.125578 26951 raft_consensus.cc:695] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Becoming Leader. State: Replica: c2333615c8734156a49767c0087e8a42, State: Running, Role: LEADER
I20250626 02:04:19.125715 26951 consensus_queue.cc:237] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:19.127125 26285 catalog_manager.cc:5582] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 reported cstate change: term changed from 1 to 2, leader changed from 56533e22a80a4098912aa35714c0da66 (127.19.220.131) to c2333615c8734156a49767c0087e8a42 (127.19.220.129). New cstate: current_term: 2 leader_uuid: "c2333615c8734156a49767c0087e8a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:19.237298 26961 raft_consensus.cc:491] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:19.237440 26961 raft_consensus.cc:513] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.237933 26961 leader_election.cc:290] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605), 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233)
I20250626 02:04:19.241282 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:19.241333 26753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2333615c8734156a49767c0087e8a42" is_pre_election: true
I20250626 02:04:19.241393 26835 raft_consensus.cc:2466] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 1.
I20250626 02:04:19.241554 26753 raft_consensus.cc:2408] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56533e22a80a4098912aa35714c0da66 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250626 02:04:19.241714 26876 leader_election.cc:304] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27; no voters:
I20250626 02:04:19.241904 26961 raft_consensus.cc:2802] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 02:04:19.241958 26961 raft_consensus.cc:491] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:19.241986 26961 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.242895 26961 raft_consensus.cc:513] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.243223 26961 leader_election.cc:290] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 2 election: Requested vote from peers c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605), 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233)
I20250626 02:04:19.243492 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:19.243501 26753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2333615c8734156a49767c0087e8a42"
I20250626 02:04:19.243590 26835 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.243599 26753 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.244385 26835 raft_consensus.cc:2466] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 2.
I20250626 02:04:19.244465 26753 raft_consensus.cc:2408] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 56533e22a80a4098912aa35714c0da66 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250626 02:04:19.244637 26876 leader_election.cc:304] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27; no voters:
I20250626 02:04:19.244786 26961 raft_consensus.cc:2802] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:04:19.244961 26961 raft_consensus.cc:695] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 2 LEADER]: Becoming Leader. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Running, Role: LEADER
I20250626 02:04:19.245122 26961 consensus_queue.cc:237] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.246426 26285 catalog_manager.cc:5582] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 reported cstate change: term changed from 1 to 2, leader changed from c2333615c8734156a49767c0087e8a42 (127.19.220.129) to 56533e22a80a4098912aa35714c0da66 (127.19.220.131). New cstate: current_term: 2 leader_uuid: "56533e22a80a4098912aa35714c0da66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:19.296777 26951 raft_consensus.cc:491] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:19.296945 26951 raft_consensus.cc:513] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.297434 26951 leader_election.cc:290] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:19.297847 26914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22af9a7f4a564cb7b19bf75222040896" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66" is_pre_election: true
I20250626 02:04:19.297849 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22af9a7f4a564cb7b19bf75222040896" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:19.297986 26914 raft_consensus.cc:2466] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 1.
I20250626 02:04:19.298017 26835 raft_consensus.cc:2466] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 1.
I20250626 02:04:19.298297 26716 leader_election.cc:304] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, c2333615c8734156a49767c0087e8a42; no voters:
I20250626 02:04:19.298462 26951 raft_consensus.cc:2802] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 02:04:19.298528 26951 raft_consensus.cc:491] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:19.298557 26951 raft_consensus.cc:3058] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.299374 26951 raft_consensus.cc:513] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.299697 26951 leader_election.cc:290] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 election: Requested vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:19.300012 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22af9a7f4a564cb7b19bf75222040896" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:19.300014 26914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22af9a7f4a564cb7b19bf75222040896" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66"
I20250626 02:04:19.300122 26835 raft_consensus.cc:3058] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.300153 26914 raft_consensus.cc:3058] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.300913 26835 raft_consensus.cc:2466] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 2.
I20250626 02:04:19.300931 26914 raft_consensus.cc:2466] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 2.
I20250626 02:04:19.301198 26713 leader_election.cc:304] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters:
I20250626 02:04:19.301339 26951 raft_consensus.cc:2802] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:04:19.301443 26951 raft_consensus.cc:695] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Becoming Leader. State: Replica: c2333615c8734156a49767c0087e8a42, State: Running, Role: LEADER
I20250626 02:04:19.301587 26951 consensus_queue.cc:237] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.302942 26951 raft_consensus.cc:491] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:19.303020 26951 raft_consensus.cc:513] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.303092 26285 catalog_manager.cc:5582] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c2333615c8734156a49767c0087e8a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:19.303316 26951 leader_election.cc:290] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:19.303643 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86b7798500d4ebea91b6d389c68ed1b" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:19.303632 26914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86b7798500d4ebea91b6d389c68ed1b" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66" is_pre_election: true
I20250626 02:04:19.303761 26835 raft_consensus.cc:2466] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 1.
I20250626 02:04:19.303790 26914 raft_consensus.cc:2408] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2333615c8734156a49767c0087e8a42 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250626 02:04:19.304035 26713 leader_election.cc:304] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters: 56533e22a80a4098912aa35714c0da66
I20250626 02:04:19.304410 26951 raft_consensus.cc:2802] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 02:04:19.304466 26951 raft_consensus.cc:491] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:19.304937 26951 raft_consensus.cc:3058] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.305773 26951 raft_consensus.cc:513] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.306033 26951 leader_election.cc:290] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 election: Requested vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:19.306308 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86b7798500d4ebea91b6d389c68ed1b" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:19.306350 26914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c86b7798500d4ebea91b6d389c68ed1b" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66"
I20250626 02:04:19.306419 26835 raft_consensus.cc:3058] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.306430 26914 raft_consensus.cc:3058] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:04:19.307142 26835 raft_consensus.cc:2466] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 2.
I20250626 02:04:19.307186 26914 raft_consensus.cc:2408] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c2333615c8734156a49767c0087e8a42 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250626 02:04:19.307379 26713 leader_election.cc:304] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters:
I20250626 02:04:19.307523 26951 raft_consensus.cc:2802] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:04:19.307615 26951 raft_consensus.cc:695] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Becoming Leader. State: Replica: c2333615c8734156a49767c0087e8a42, State: Running, Role: LEADER
I20250626 02:04:19.307752 26951 consensus_queue.cc:237] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.309191 26285 catalog_manager.cc:5582] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 reported cstate change: term changed from 1 to 2, leader changed from 56533e22a80a4098912aa35714c0da66 (127.19.220.131) to c2333615c8734156a49767c0087e8a42 (127.19.220.129). New cstate: current_term: 2 leader_uuid: "c2333615c8734156a49767c0087e8a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:19.407337 26951 raft_consensus.cc:491] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:19.407486 26951 raft_consensus.cc:513] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.407882 26951 leader_election.cc:290] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:19.408339 26914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66" is_pre_election: true
I20250626 02:04:19.408340 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:19.408529 26835 raft_consensus.cc:2466] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 2.
I20250626 02:04:19.408814 26713 leader_election.cc:304] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters: 56533e22a80a4098912aa35714c0da66
I20250626 02:04:19.408993 26951 raft_consensus.cc:2802] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 02:04:19.409050 26951 raft_consensus.cc:491] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:19.409081 26951 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:04:19.409987 26951 raft_consensus.cc:513] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.410313 26951 leader_election.cc:290] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 3 election: Requested vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:19.410648 26914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "56533e22a80a4098912aa35714c0da66"
I20250626 02:04:19.410648 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "292cdbfda66145d98d69d39f4811ed37" candidate_uuid: "c2333615c8734156a49767c0087e8a42" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:19.410808 26835 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:04:19.411649 26835 raft_consensus.cc:2466] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2333615c8734156a49767c0087e8a42 in term 3.
I20250626 02:04:19.411860 26713 leader_election.cc:304] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74cf8df2c3aa488195da158f4be08d27, c2333615c8734156a49767c0087e8a42; no voters: 56533e22a80a4098912aa35714c0da66
I20250626 02:04:19.412005 26951 raft_consensus.cc:2802] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:04:19.412166 26951 raft_consensus.cc:695] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 3 LEADER]: Becoming Leader. State: Replica: c2333615c8734156a49767c0087e8a42, State: Running, Role: LEADER
I20250626 02:04:19.412335 26951 consensus_queue.cc:237] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.413738 26285 catalog_manager.cc:5582] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 reported cstate change: term changed from 2 to 3, leader changed from 56533e22a80a4098912aa35714c0da66 (127.19.220.131) to c2333615c8734156a49767c0087e8a42 (127.19.220.129). New cstate: current_term: 3 leader_uuid: "c2333615c8734156a49767c0087e8a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:19.518397 26961 raft_consensus.cc:491] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:19.518579 26961 raft_consensus.cc:513] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:19.519002 26961 leader_election.cc:290] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605)
I20250626 02:04:19.519443 26753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c2333615c8734156a49767c0087e8a42" is_pre_election: true
I20250626 02:04:19.519443 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27" is_pre_election: true
I20250626 02:04:19.519797 26835 raft_consensus.cc:2466] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 2.
I20250626 02:04:19.520051 26876 leader_election.cc:304] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27; no voters: c2333615c8734156a49767c0087e8a42
I20250626 02:04:19.520294 26961 raft_consensus.cc:2802] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 02:04:19.520362 26961 raft_consensus.cc:491] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:19.520396 26961 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:04:19.521291 26961 raft_consensus.cc:513] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:19.521649 26961 leader_election.cc:290] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 3 election: Requested vote from peers 74cf8df2c3aa488195da158f4be08d27 (127.19.220.130:33233), c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605)
I20250626 02:04:19.521970 26835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74cf8df2c3aa488195da158f4be08d27"
I20250626 02:04:19.521968 26753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b50e1c4d5545a7aa7fc1c8bc1d035f" candidate_uuid: "56533e22a80a4098912aa35714c0da66" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c2333615c8734156a49767c0087e8a42"
I20250626 02:04:19.522083 26835 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:04:19.522913 26835 raft_consensus.cc:2466] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56533e22a80a4098912aa35714c0da66 in term 3.
I20250626 02:04:19.523157 26876 leader_election.cc:304] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56533e22a80a4098912aa35714c0da66, 74cf8df2c3aa488195da158f4be08d27; no voters: c2333615c8734156a49767c0087e8a42
I20250626 02:04:19.523305 26961 raft_consensus.cc:2802] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 02:04:19.523392 26961 raft_consensus.cc:695] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 3 LEADER]: Becoming Leader. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Running, Role: LEADER
I20250626 02:04:19.523521 26961 consensus_queue.cc:237] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:19.525125 26285 catalog_manager.cc:5582] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 reported cstate change: term changed from 2 to 3, leader changed from c2333615c8734156a49767c0087e8a42 (127.19.220.129) to 56533e22a80a4098912aa35714c0da66 (127.19.220.131). New cstate: current_term: 3 leader_uuid: "56533e22a80a4098912aa35714c0da66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:19.593255 26835 raft_consensus.cc:1238] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Rejecting Update request from peer c2333615c8734156a49767c0087e8a42 for earlier term 2. Current term is 3. Ops: []
I20250626 02:04:19.593755 26951 consensus_queue.cc:1046] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:19.594017 26955 raft_consensus.cc:3053] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:04:19.594094 26955 raft_consensus.cc:738] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c2333615c8734156a49767c0087e8a42, State: Running, Role: LEADER
I20250626 02:04:19.594254 26955 consensus_queue.cc:260] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
I20250626 02:04:19.594416 26955 raft_consensus.cc:3058] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:04:19.672317 26753 raft_consensus.cc:1238] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 3 LEADER]: Rejecting Update request from peer 56533e22a80a4098912aa35714c0da66 for earlier term 2. Current term is 3. Ops: []
I20250626 02:04:19.672752 26961 consensus_queue.cc:1046] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:19.672925 26961 consensus_queue.cc:1230] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [LEADER]: Peer c2333615c8734156a49767c0087e8a42 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250626 02:04:19.673008 26964 raft_consensus.cc:3053] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 02:04:19.673063 26964 raft_consensus.cc:738] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 56533e22a80a4098912aa35714c0da66, State: Running, Role: LEADER
I20250626 02:04:19.673219 26964 consensus_queue.cc:260] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:19.673388 26964 raft_consensus.cc:3058] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Advancing to term 3
I20250626 02:04:19.777019 26914 raft_consensus.cc:1273] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250626 02:04:19.777122 26914 pending_rounds.cc:85] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Aborting all ops after (but not including) 0
I20250626 02:04:19.777146 26914 pending_rounds.cc:107] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250626 02:04:19.777168 26914 raft_consensus.cc:2887] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 02:04:19.777583 26955 consensus_queue.cc:1035] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:19.780038 26835 raft_consensus.cc:1273] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 02:04:19.780370 26955 consensus_queue.cc:1035] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:19.831476 26914 raft_consensus.cc:1273] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 3 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 02:04:19.831933 26951 consensus_queue.cc:1035] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:19.832158 26951 consensus_queue.cc:1230] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [LEADER]: Peer 56533e22a80a4098912aa35714c0da66 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250626 02:04:19.832479 26835 raft_consensus.cc:1273] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:04:19.832875 26951 consensus_queue.cc:1035] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:19.832984 26914 pending_rounds.cc:85] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Aborting all ops after (but not including) 0
I20250626 02:04:19.833055 26914 pending_rounds.cc:107] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250626 02:04:19.833102 26914 raft_consensus.cc:2887] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 02:04:19.835027 26834 raft_consensus.cc:1273] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 02:04:19.835350 26955 consensus_queue.cc:1035] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:19.837666 26914 raft_consensus.cc:1273] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:04:19.838245 26978 consensus_queue.cc:1035] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 02:04:19.871565 26334 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20250626 02:04:19.872462 26334 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f, TS c2333615c8734156a49767c0087e8a42 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
W20250626 02:04:19.872529 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f, TS c2333615c8734156a49767c0087e8a42 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } }
W20250626 02:04:19.873328 26334 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 22af9a7f4a564cb7b19bf75222040896, TS 56533e22a80a4098912aa35714c0da66 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
W20250626 02:04:19.873371 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 22af9a7f4a564cb7b19bf75222040896, TS 56533e22a80a4098912aa35714c0da66 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
W20250626 02:04:19.874116 26334 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c86b7798500d4ebea91b6d389c68ed1b, TS 74cf8df2c3aa488195da158f4be08d27 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
W20250626 02:04:19.874161 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet c86b7798500d4ebea91b6d389c68ed1b, TS 74cf8df2c3aa488195da158f4be08d27 -> TS 933876bfc7a4435ca9dd1a4feab14528 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } }
I20250626 02:04:20.125676 26834 raft_consensus.cc:1273] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Refusing update from remote peer 56533e22a80a4098912aa35714c0da66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 02:04:20.126111 26964 consensus_queue.cc:1035] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:20.129266 26753 raft_consensus.cc:1273] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: Refusing update from remote peer 56533e22a80a4098912aa35714c0da66: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 02:04:20.129668 26964 consensus_queue.cc:1035] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:20.129818 26964 consensus_queue.cc:1230] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [LEADER]: Peer c2333615c8734156a49767c0087e8a42 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250626 02:04:20.130437 26753 pending_rounds.cc:85] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Aborting all ops after (but not including) 0
I20250626 02:04:20.130496 26753 pending_rounds.cc:107] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250626 02:04:20.130527 26753 raft_consensus.cc:2887] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 02:04:20.876649 26914 consensus_queue.cc:237] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
I20250626 02:04:20.878005 26753 raft_consensus.cc:1273] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: Refusing update from remote peer 56533e22a80a4098912aa35714c0da66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 02:04:20.878211 26834 raft_consensus.cc:1273] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Refusing update from remote peer 56533e22a80a4098912aa35714c0da66: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 02:04:20.878392 26984 consensus_queue.cc:1035] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:20.878583 26964 consensus_queue.cc:1035] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:20.880172 26964 raft_consensus.cc:2953] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.880506 26834 raft_consensus.cc:2953] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.880580 26753 raft_consensus.cc:2953] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.882766 26285 catalog_manager.cc:5582] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 reported cstate change: config changed from index -1 to 3, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New cstate: current_term: 3 leader_uuid: "56533e22a80a4098912aa35714c0da66" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:20.883299 26753 consensus_queue.cc:237] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
I20250626 02:04:20.884718 26834 raft_consensus.cc:1273] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250626 02:04:20.885023 26879 consensus_peers.cc:487] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 -> Peer 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132:46095): Couldn't send request to peer 933876bfc7a4435ca9dd1a4feab14528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b50e1c4d5545a7aa7fc1c8bc1d035f. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:20.885136 26978 consensus_queue.cc:1035] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:20.885253 26914 raft_consensus.cc:1273] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 02:04:20.885576 26978 consensus_queue.cc:1035] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:20.887248 26986 raft_consensus.cc:2953] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.887632 26914 raft_consensus.cc:2953] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.887845 26834 raft_consensus.cc:2953] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.889819 26284 catalog_manager.cc:5582] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 reported cstate change: config changed from index -1 to 3, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New cstate: current_term: 2 leader_uuid: "c2333615c8734156a49767c0087e8a42" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.890029 26753 consensus_queue.cc:237] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } attrs { replace: true } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
W20250626 02:04:20.891124 26716 consensus_peers.cc:487] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 -> Peer 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132:46095): Couldn't send request to peer 933876bfc7a4435ca9dd1a4feab14528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22af9a7f4a564cb7b19bf75222040896. This is attempt 1: this message will repeat every 5th retry.
W20250626 02:04:20.891446 26716 consensus_peers.cc:487] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 -> Peer 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132:46095): Couldn't send request to peer 933876bfc7a4435ca9dd1a4feab14528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c86b7798500d4ebea91b6d389c68ed1b. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:20.891453 26914 raft_consensus.cc:1273] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:04:20.891666 26834 raft_consensus.cc:1273] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Refusing update from remote peer c2333615c8734156a49767c0087e8a42: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 02:04:20.891832 26986 consensus_queue.cc:1035] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:04:20.891990 26951 consensus_queue.cc:1035] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:04:20.893577 26986 raft_consensus.cc:2953] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } attrs { replace: true } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.893925 26914 raft_consensus.cc:2953] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } attrs { replace: true } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.895067 26834 raft_consensus.cc:2953] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } attrs { replace: true } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:20.895757 26286 catalog_manager.cc:5582] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 reported cstate change: config changed from index -1 to 2, NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 (127.19.220.132) added. New cstate: current_term: 2 leader_uuid: "c2333615c8734156a49767c0087e8a42" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } attrs { replace: true } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } } }
I20250626 02:04:21.261801 27001 ts_tablet_manager.cc:927] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: Initiating tablet copy from peer 56533e22a80a4098912aa35714c0da66 (127.19.220.131:35229)
I20250626 02:04:21.262167 27001 tablet_copy_client.cc:323] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:35229
I20250626 02:04:21.266207 26926 tablet_copy_service.cc:140] P 56533e22a80a4098912aa35714c0da66: Received BeginTabletCopySession request for tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f from peer 933876bfc7a4435ca9dd1a4feab14528 ({username='slave'} at 127.0.0.1:49556)
I20250626 02:04:21.266348 26926 tablet_copy_service.cc:161] P 56533e22a80a4098912aa35714c0da66: Beginning new tablet copy session on tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f from peer 933876bfc7a4435ca9dd1a4feab14528 at {username='slave'} at 127.0.0.1:49556: session id = 933876bfc7a4435ca9dd1a4feab14528-c3b50e1c4d5545a7aa7fc1c8bc1d035f
I20250626 02:04:21.267242 26926 tablet_copy_source_session.cc:215] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:21.268049 27001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:21.270167 27001 tablet_copy_client.cc:806] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:21.270357 27001 tablet_copy_client.cc:670] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:21.271358 27001 tablet_copy_client.cc:538] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:21.272560 27001 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap starting.
I20250626 02:04:21.276064 27001 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:21.276399 27001 tablet_bootstrap.cc:492] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap complete.
I20250626 02:04:21.276569 27001 ts_tablet_manager.cc:1397] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20250626 02:04:21.277037 27001 raft_consensus.cc:357] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
I20250626 02:04:21.277175 27001 raft_consensus.cc:738] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 933876bfc7a4435ca9dd1a4feab14528, State: Initialized, Role: LEARNER
I20250626 02:04:21.277280 27001 consensus_queue.cc:260] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
I20250626 02:04:21.277637 27001 ts_tablet_manager.cc:1428] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250626 02:04:21.278311 26926 tablet_copy_service.cc:342] P 56533e22a80a4098912aa35714c0da66: Request end of tablet copy session 933876bfc7a4435ca9dd1a4feab14528-c3b50e1c4d5545a7aa7fc1c8bc1d035f received from {username='slave'} at 127.0.0.1:49556
I20250626 02:04:21.278393 26926 tablet_copy_service.cc:434] P 56533e22a80a4098912aa35714c0da66: ending tablet copy session 933876bfc7a4435ca9dd1a4feab14528-c3b50e1c4d5545a7aa7fc1c8bc1d035f on tablet c3b50e1c4d5545a7aa7fc1c8bc1d035f with peer 933876bfc7a4435ca9dd1a4feab14528
I20250626 02:04:21.420907 27001 ts_tablet_manager.cc:927] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: Initiating tablet copy from peer c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605)
I20250626 02:04:21.421289 27001 tablet_copy_client.cc:323] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:39605
I20250626 02:04:21.425122 26763 tablet_copy_service.cc:140] P c2333615c8734156a49767c0087e8a42: Received BeginTabletCopySession request for tablet 22af9a7f4a564cb7b19bf75222040896 from peer 933876bfc7a4435ca9dd1a4feab14528 ({username='slave'} at 127.0.0.1:46304)
I20250626 02:04:21.425233 26763 tablet_copy_service.cc:161] P c2333615c8734156a49767c0087e8a42: Beginning new tablet copy session on tablet 22af9a7f4a564cb7b19bf75222040896 from peer 933876bfc7a4435ca9dd1a4feab14528 at {username='slave'} at 127.0.0.1:46304: session id = 933876bfc7a4435ca9dd1a4feab14528-22af9a7f4a564cb7b19bf75222040896
I20250626 02:04:21.426180 26763 tablet_copy_source_session.cc:215] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:21.427012 27001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22af9a7f4a564cb7b19bf75222040896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:21.429098 27001 tablet_copy_client.cc:806] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:21.429244 27001 tablet_copy_client.cc:670] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:21.430222 27001 tablet_copy_client.cc:538] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:21.431458 27001 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap starting.
I20250626 02:04:21.435427 27001 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:21.435683 27001 tablet_bootstrap.cc:492] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap complete.
I20250626 02:04:21.435851 27001 ts_tablet_manager.cc:1397] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250626 02:04:21.436367 27001 raft_consensus.cc:357] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
I20250626 02:04:21.436507 27001 raft_consensus.cc:738] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 933876bfc7a4435ca9dd1a4feab14528, State: Initialized, Role: LEARNER
I20250626 02:04:21.436616 27001 consensus_queue.cc:260] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } attrs { replace: true } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
I20250626 02:04:21.436925 27001 ts_tablet_manager.cc:1428] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:21.437328 26763 tablet_copy_service.cc:342] P c2333615c8734156a49767c0087e8a42: Request end of tablet copy session 933876bfc7a4435ca9dd1a4feab14528-22af9a7f4a564cb7b19bf75222040896 received from {username='slave'} at 127.0.0.1:46304
I20250626 02:04:21.437417 26763 tablet_copy_service.cc:434] P c2333615c8734156a49767c0087e8a42: ending tablet copy session 933876bfc7a4435ca9dd1a4feab14528-22af9a7f4a564cb7b19bf75222040896 on tablet 22af9a7f4a564cb7b19bf75222040896 with peer 933876bfc7a4435ca9dd1a4feab14528
I20250626 02:04:21.502748 27001 ts_tablet_manager.cc:927] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: Initiating tablet copy from peer c2333615c8734156a49767c0087e8a42 (127.19.220.129:39605)
I20250626 02:04:21.503113 27001 tablet_copy_client.cc:323] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.129:39605
I20250626 02:04:21.503536 26763 tablet_copy_service.cc:140] P c2333615c8734156a49767c0087e8a42: Received BeginTabletCopySession request for tablet c86b7798500d4ebea91b6d389c68ed1b from peer 933876bfc7a4435ca9dd1a4feab14528 ({username='slave'} at 127.0.0.1:46304)
I20250626 02:04:21.503643 26763 tablet_copy_service.cc:161] P c2333615c8734156a49767c0087e8a42: Beginning new tablet copy session on tablet c86b7798500d4ebea91b6d389c68ed1b from peer 933876bfc7a4435ca9dd1a4feab14528 at {username='slave'} at 127.0.0.1:46304: session id = 933876bfc7a4435ca9dd1a4feab14528-c86b7798500d4ebea91b6d389c68ed1b
I20250626 02:04:21.504858 26763 tablet_copy_source_session.cc:215] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:21.505748 27001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86b7798500d4ebea91b6d389c68ed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:21.507944 27001 tablet_copy_client.cc:806] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:21.508167 27001 tablet_copy_client.cc:670] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:21.509166 27001 tablet_copy_client.cc:538] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:21.510247 27001 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap starting.
I20250626 02:04:21.513337 27001 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:21.513568 27001 tablet_bootstrap.cc:492] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: Bootstrap complete.
I20250626 02:04:21.513731 27001 ts_tablet_manager.cc:1397] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250626 02:04:21.514106 27001 raft_consensus.cc:357] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } attrs { replace: true } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
I20250626 02:04:21.514211 27001 raft_consensus.cc:738] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 933876bfc7a4435ca9dd1a4feab14528, State: Initialized, Role: LEARNER
I20250626 02:04:21.514330 27001 consensus_queue.cc:260] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c2333615c8734156a49767c0087e8a42" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 39605 } } peers { permanent_uuid: "74cf8df2c3aa488195da158f4be08d27" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 33233 } attrs { replace: true } } peers { permanent_uuid: "56533e22a80a4098912aa35714c0da66" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 35229 } } peers { permanent_uuid: "933876bfc7a4435ca9dd1a4feab14528" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 46095 } attrs { promote: true } }
I20250626 02:04:21.514637 27001 ts_tablet_manager.cc:1428] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:21.515090 26763 tablet_copy_service.cc:342] P c2333615c8734156a49767c0087e8a42: Request end of tablet copy session 933876bfc7a4435ca9dd1a4feab14528-c86b7798500d4ebea91b6d389c68ed1b received from {username='slave'} at 127.0.0.1:46304
I20250626 02:04:21.515168 26763 tablet_copy_service.cc:434] P c2333615c8734156a49767c0087e8a42: ending tablet copy session 933876bfc7a4435ca9dd1a4feab14528-c86b7798500d4ebea91b6d389c68ed1b on tablet c86b7798500d4ebea91b6d389c68ed1b with peer 933876bfc7a4435ca9dd1a4feab14528
I20250626 02:04:21.675467 26660 raft_consensus.cc:1215] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250626 02:04:21.902427 26660 raft_consensus.cc:1215] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250626 02:04:21.942396 26660 raft_consensus.cc:1215] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250626 02:04:22.111416 20338 tablet_server.cc:178] TabletServer@127.19.220.129:39605 shutting down...
W20250626 02:04:22.112709 26331 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 11 similar messages]
W20250626 02:04:22.113569 26334 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:04:22.113642 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:04:22.115538 26334 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:04:22.115614 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250626 02:04:22.115749 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:22.115900 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.116041 20338 raft_consensus.cc:2241] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 3 LEADER]: Raft consensus shutting down.
I20250626 02:04:22.116281 20338 raft_consensus.cc:2270] T 292cdbfda66145d98d69d39f4811ed37 P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.116832 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.116971 20338 raft_consensus.cc:2241] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:04:22.117161 20338 raft_consensus.cc:2270] T c86b7798500d4ebea91b6d389c68ed1b P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.117550 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.117647 20338 raft_consensus.cc:2241] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:22.117758 20338 raft_consensus.cc:2270] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P c2333615c8734156a49767c0087e8a42 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.118043 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.118132 20338 raft_consensus.cc:2241] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:04:22.118295 20338 raft_consensus.cc:2270] T 22af9a7f4a564cb7b19bf75222040896 P c2333615c8734156a49767c0087e8a42 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.121670 20338 tablet_server.cc:195] TabletServer@127.19.220.129:39605 shutdown complete.
I20250626 02:04:22.123965 20338 tablet_server.cc:178] TabletServer@127.19.220.130:33233 shutting down...
I20250626 02:04:22.129509 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:22.129658 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.129778 20338 raft_consensus.cc:2241] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:22.129900 20338 raft_consensus.cc:2270] T 292cdbfda66145d98d69d39f4811ed37 P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.130312 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.130388 20338 raft_consensus.cc:2241] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:22.130502 20338 raft_consensus.cc:2270] T c86b7798500d4ebea91b6d389c68ed1b P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.130870 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.130975 20338 raft_consensus.cc:2241] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:22.131072 20338 raft_consensus.cc:2270] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 74cf8df2c3aa488195da158f4be08d27 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.131390 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.131479 20338 raft_consensus.cc:2241] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:22.131580 20338 raft_consensus.cc:2270] T 22af9a7f4a564cb7b19bf75222040896 P 74cf8df2c3aa488195da158f4be08d27 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.144100 20338 tablet_server.cc:195] TabletServer@127.19.220.130:33233 shutdown complete.
I20250626 02:04:22.146481 20338 tablet_server.cc:178] TabletServer@127.19.220.131:35229 shutting down...
W20250626 02:04:22.147301 26334 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 02:04:22.147393 26334 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250626 02:04:22.150599 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:22.150738 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.150856 20338 raft_consensus.cc:2241] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:22.150969 20338 raft_consensus.cc:2270] T 292cdbfda66145d98d69d39f4811ed37 P 56533e22a80a4098912aa35714c0da66 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.151335 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.151464 20338 raft_consensus.cc:2241] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:22.151609 20338 raft_consensus.cc:2270] T c86b7798500d4ebea91b6d389c68ed1b P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.151973 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.152070 20338 raft_consensus.cc:2241] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 3 LEADER]: Raft consensus shutting down.
I20250626 02:04:22.152297 27013 raft_consensus.cc:1062] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: attempting to promote NON_VOTER 933876bfc7a4435ca9dd1a4feab14528 to VOTER
I20250626 02:04:22.152357 20338 raft_consensus.cc:2270] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66 [term 3 FOLLOWER]: Raft consensus is shut down!
W20250626 02:04:22.152412 27013 raft_consensus.cc:1065] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 56533e22a80a4098912aa35714c0da66: Unable to promote non-voter 933876bfc7a4435ca9dd1a4feab14528: Illegal state: RaftConsensus is not running: State = Stopped
I20250626 02:04:22.152755 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:22.152819 20338 raft_consensus.cc:2241] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:22.152894 20338 raft_consensus.cc:2270] T 22af9a7f4a564cb7b19bf75222040896 P 56533e22a80a4098912aa35714c0da66 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.155264 20338 tablet_server.cc:195] TabletServer@127.19.220.131:35229 shutdown complete.
I20250626 02:04:22.157130 20338 tablet_server.cc:178] TabletServer@127.19.220.132:0 shutting down...
I20250626 02:04:22.161654 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:22.161804 20338 tablet_replica.cc:331] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528: stopping tablet replica
I20250626 02:04:22.161914 20338 raft_consensus.cc:2241] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Raft consensus shutting down.
I20250626 02:04:22.161991 20338 raft_consensus.cc:2270] T c86b7798500d4ebea91b6d389c68ed1b P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Raft consensus is shut down!
I20250626 02:04:22.162277 20338 tablet_replica.cc:331] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528: stopping tablet replica
I20250626 02:04:22.162346 20338 raft_consensus.cc:2241] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528 [term 3 LEARNER]: Raft consensus shutting down.
I20250626 02:04:22.162410 20338 raft_consensus.cc:2270] T c3b50e1c4d5545a7aa7fc1c8bc1d035f P 933876bfc7a4435ca9dd1a4feab14528 [term 3 LEARNER]: Raft consensus is shut down!
I20250626 02:04:22.162667 20338 tablet_replica.cc:331] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528: stopping tablet replica
I20250626 02:04:22.162726 20338 raft_consensus.cc:2241] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Raft consensus shutting down.
I20250626 02:04:22.162788 20338 raft_consensus.cc:2270] T 22af9a7f4a564cb7b19bf75222040896 P 933876bfc7a4435ca9dd1a4feab14528 [term 2 LEARNER]: Raft consensus is shut down!
I20250626 02:04:22.174995 20338 tablet_server.cc:195] TabletServer@127.19.220.132:0 shutdown complete.
I20250626 02:04:22.176887 20338 master.cc:561] Master@127.19.220.190:42575 shutting down...
I20250626 02:04:22.179971 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:22.180102 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:22.180222 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P db81ca31d2ea44b1ad8641078447b44b: stopping tablet replica
I20250626 02:04:22.192682 20338 master.cc:583] Master@127.19.220.190:42575 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11427 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20250626 02:04:22.201211 20338 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.220.190:42333
I20250626 02:04:22.201548 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:22.203119 27020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:22.203119 27018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:22.203245 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:22.203238 27017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:22.203766 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:22.203825 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:22.203855 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903462203855 us; error 0 us; skew 500 ppm
I20250626 02:04:22.204731 20338 webserver.cc:469] Webserver started at http://127.19.220.190:41241/ using document root <none> and password file <none>
I20250626 02:04:22.204850 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:22.204890 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:22.204943 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:22.205271 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/master-0-root/instance:
uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:22.206626 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:22.207446 27025 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.207697 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:22.207790 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/master-0-root
uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:22.207861 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:22.218170 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:22.218518 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:22.222481 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.190:42333
I20250626 02:04:22.225734 27077 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.190:42333 every 8 connection(s)
I20250626 02:04:22.226138 27078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:22.227897 27078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: Bootstrap starting.
I20250626 02:04:22.228652 27078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:22.229538 27078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: No bootstrap required, opened a new log
I20250626 02:04:22.229887 27078 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f" member_type: VOTER }
I20250626 02:04:22.229969 27078 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:22.229990 27078 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed32e6b9b6945d9a52be6717c9f7f8f, State: Initialized, Role: FOLLOWER
I20250626 02:04:22.230084 27078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f" member_type: VOTER }
I20250626 02:04:22.230136 27078 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:04:22.230162 27078 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:04:22.230195 27078 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:22.230806 27078 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f" member_type: VOTER }
I20250626 02:04:22.230919 27078 leader_election.cc:304] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ed32e6b9b6945d9a52be6717c9f7f8f; no voters:
I20250626 02:04:22.231050 27078 leader_election.cc:290] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:04:22.231166 27081 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:22.231340 27081 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 1 LEADER]: Becoming Leader. State: Replica: 8ed32e6b9b6945d9a52be6717c9f7f8f, State: Running, Role: LEADER
I20250626 02:04:22.231434 27078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:04:22.231453 27081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f" member_type: VOTER }
I20250626 02:04:22.231909 27083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ed32e6b9b6945d9a52be6717c9f7f8f. Latest consensus state: current_term: 1 leader_uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f" member_type: VOTER } }
I20250626 02:04:22.231894 27082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed32e6b9b6945d9a52be6717c9f7f8f" member_type: VOTER } }
I20250626 02:04:22.232023 27083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [sys.catalog]: This master's current role is: LEADER
I20250626 02:04:22.232065 27082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [sys.catalog]: This master's current role is: LEADER
I20250626 02:04:22.232344 27087 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:04:22.233235 27087 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:04:22.233476 20338 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 02:04:22.234972 27087 catalog_manager.cc:1349] Generated new cluster ID: 9684dc89f5c348d483546bb4b882c637
I20250626 02:04:22.235014 27087 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:04:22.245576 27087 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:04:22.246063 27087 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:04:22.253261 27087 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: Generated new TSK 0
I20250626 02:04:22.253405 27087 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:04:22.265738 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:22.267493 27099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:22.267611 27102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:22.267664 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:22.267699 27100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:22.267968 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:22.268010 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:22.268025 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903462268025 us; error 0 us; skew 500 ppm
I20250626 02:04:22.268963 20338 webserver.cc:469] Webserver started at http://127.19.220.129:40835/ using document root <none> and password file <none>
I20250626 02:04:22.269106 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:22.269156 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:22.269236 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:22.269644 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root/instance:
uuid: "7b20a67f201e4924a8f003667eeb604b"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:22.271075 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:22.271870 27107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.272116 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:22.272203 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "7b20a67f201e4924a8f003667eeb604b"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:22.272256 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:22.293275 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:22.293601 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:22.293838 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:22.294271 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:22.294308 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.294339 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:22.294373 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.298168 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:37353
I20250626 02:04:22.298233 27170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:37353 every 8 connection(s)
I20250626 02:04:22.298964 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:22.300686 27175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:22.300899 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:22.300735 27176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:22.300899 27178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:22.301255 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:22.301301 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:22.301323 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903462301322 us; error 0 us; skew 500 ppm
I20250626 02:04:22.302294 20338 webserver.cc:469] Webserver started at http://127.19.220.130:46053/ using document root <none> and password file <none>
I20250626 02:04:22.302441 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:22.302508 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:22.302589 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:22.302960 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root/instance:
uuid: "d26ebacebabe43d6822b5a63ff17c6e2"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:22.303854 27171 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42333
I20250626 02:04:22.303946 27171 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:22.304204 27171 heartbeater.cc:507] Master 127.19.220.190:42333 requested a full tablet report, sending...
I20250626 02:04:22.304754 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:22.304833 27042 ts_manager.cc:194] Registered new tserver with Master: 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353)
I20250626 02:04:22.305558 27042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55576
I20250626 02:04:22.305648 27183 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.305914 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:22.305981 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root
uuid: "d26ebacebabe43d6822b5a63ff17c6e2"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:22.306046 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:22.317114 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:22.317442 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:22.317711 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:22.318181 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:22.318218 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.318248 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:22.318267 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.322113 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.130:35589
I20250626 02:04:22.322166 27246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.130:35589 every 8 connection(s)
I20250626 02:04:22.322901 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:22.324539 27250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:22.324643 27251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:22.324736 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:22.324834 27253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:22.325114 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:22.325156 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:22.325196 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903462325195 us; error 0 us; skew 500 ppm
I20250626 02:04:22.326151 20338 webserver.cc:469] Webserver started at http://127.19.220.131:46009/ using document root <none> and password file <none>
I20250626 02:04:22.326311 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:22.326375 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:22.326499 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:22.326910 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root/instance:
uuid: "822f01a34c91459993dd88632e7cfe41"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:22.326985 27247 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42333
I20250626 02:04:22.327080 27247 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:22.327275 27247 heartbeater.cc:507] Master 127.19.220.190:42333 requested a full tablet report, sending...
I20250626 02:04:22.327827 27042 ts_manager.cc:194] Registered new tserver with Master: d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:22.328359 27042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55580
I20250626 02:04:22.328537 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:22.329506 27258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.329787 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:22.329843 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root
uuid: "822f01a34c91459993dd88632e7cfe41"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:22.329896 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:22.341449 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:22.341799 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:22.342033 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:22.342482 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:22.342523 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.342553 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:22.342568 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:22.346680 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.131:41677
I20250626 02:04:22.346742 27321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.131:41677 every 8 connection(s)
I20250626 02:04:22.351027 27322 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42333
I20250626 02:04:22.351114 27322 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:22.351312 27322 heartbeater.cc:507] Master 127.19.220.190:42333 requested a full tablet report, sending...
I20250626 02:04:22.351857 27042 ts_manager.cc:194] Registered new tserver with Master: 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:22.352377 27042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55592
I20250626 02:04:22.352639 20338 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005588081s
I20250626 02:04:23.306563 27171 heartbeater.cc:499] Master 127.19.220.190:42333 was elected leader, sending a full tablet report...
I20250626 02:04:23.329281 27247 heartbeater.cc:499] Master 127.19.220.190:42333 was elected leader, sending a full tablet report...
I20250626 02:04:23.353252 27322 heartbeater.cc:499] Master 127.19.220.190:42333 was elected leader, sending a full tablet report...
I20250626 02:04:23.378583 20338 test_util.cc:276] Using random seed: -84377964
I20250626 02:04:23.389679 27042 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38530:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:04:23.390228 27042 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:04:23.400465 27209 tablet_service.cc:1468] Processing CreateTablet for tablet ef6548b73bbc43229fc8f974b4165ac7 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:23.400658 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 4cf3aa09af90400190885a70f7c47362 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:23.400808 27209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6548b73bbc43229fc8f974b4165ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.400986 27286 tablet_service.cc:1468] Processing CreateTablet for tablet 06faf0dbd13448d4a2bf7bb4edfce338 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:23.401039 27208 tablet_service.cc:1468] Processing CreateTablet for tablet 72a83458cfda49a6a0fbff57a1371c22 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:23.401212 27286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06faf0dbd13448d4a2bf7bb4edfce338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.401284 27208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a83458cfda49a6a0fbff57a1371c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.403044 27342 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41: Bootstrap starting.
I20250626 02:04:23.403204 27285 tablet_service.cc:1468] Processing CreateTablet for tablet 4cf3aa09af90400190885a70f7c47362 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:23.403311 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf3aa09af90400190885a70f7c47362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.403411 27285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf3aa09af90400190885a70f7c47362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.400465 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 06faf0dbd13448d4a2bf7bb4edfce338 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:23.403688 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06faf0dbd13448d4a2bf7bb4edfce338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.404012 27342 tablet_bootstrap.cc:654] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.404986 27283 tablet_service.cc:1468] Processing CreateTablet for tablet 72a83458cfda49a6a0fbff57a1371c22 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:23.405225 27283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a83458cfda49a6a0fbff57a1371c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.405969 27343 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: Bootstrap starting.
I20250626 02:04:23.406697 27343 tablet_bootstrap.cc:654] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.407137 27284 tablet_service.cc:1468] Processing CreateTablet for tablet ef6548b73bbc43229fc8f974b4165ac7 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:23.407369 27284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6548b73bbc43229fc8f974b4165ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.407670 27343 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: No bootstrap required, opened a new log
I20250626 02:04:23.407739 27343 ts_tablet_manager.cc:1397] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:04:23.408056 27343 raft_consensus.cc:357] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.408126 27343 raft_consensus.cc:383] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.408188 27343 raft_consensus.cc:738] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.408293 27343 consensus_queue.cc:260] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.408720 27343 ts_tablet_manager.cc:1428] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:23.408893 27343 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2: Bootstrap starting.
I20250626 02:04:23.409649 27342 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41: No bootstrap required, opened a new log
I20250626 02:04:23.409736 27342 ts_tablet_manager.cc:1397] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250626 02:04:23.409843 27343 tablet_bootstrap.cc:654] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.410194 27342 raft_consensus.cc:357] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.410300 27342 raft_consensus.cc:383] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.410339 27342 raft_consensus.cc:738] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.410475 27342 consensus_queue.cc:260] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.410823 27342 ts_tablet_manager.cc:1428] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:23.411003 27342 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: Bootstrap starting.
I20250626 02:04:23.411540 27343 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2: No bootstrap required, opened a new log
I20250626 02:04:23.411620 27343 ts_tablet_manager.cc:1397] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250626 02:04:23.411919 27342 tablet_bootstrap.cc:654] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.411945 27135 tablet_service.cc:1468] Processing CreateTablet for tablet 06faf0dbd13448d4a2bf7bb4edfce338 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:23.412041 27343 raft_consensus.cc:357] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.412122 27343 raft_consensus.cc:383] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.412184 27343 raft_consensus.cc:738] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.412225 27135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06faf0dbd13448d4a2bf7bb4edfce338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.412317 27343 consensus_queue.cc:260] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.412566 27343 ts_tablet_manager.cc:1428] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:23.413291 27342 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: No bootstrap required, opened a new log
I20250626 02:04:23.413362 27342 ts_tablet_manager.cc:1397] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:23.413671 27342 raft_consensus.cc:357] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.413774 27342 raft_consensus.cc:383] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.413789 27343 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: Bootstrap starting.
I20250626 02:04:23.413810 27342 raft_consensus.cc:738] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.413916 27342 consensus_queue.cc:260] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.414150 27342 ts_tablet_manager.cc:1428] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.414326 27342 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41: Bootstrap starting.
I20250626 02:04:23.414475 27350 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:23.414763 27343 tablet_bootstrap.cc:654] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.414695 27134 tablet_service.cc:1468] Processing CreateTablet for tablet 4cf3aa09af90400190885a70f7c47362 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:23.414943 27134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf3aa09af90400190885a70f7c47362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.416091 27350 tablet_bootstrap.cc:654] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.417558 27343 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: No bootstrap required, opened a new log
I20250626 02:04:23.417646 27343 ts_tablet_manager.cc:1397] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250626 02:04:23.416340 27133 tablet_service.cc:1468] Processing CreateTablet for tablet ef6548b73bbc43229fc8f974b4165ac7 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:23.416926 27342 tablet_bootstrap.cc:654] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.418041 27133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6548b73bbc43229fc8f974b4165ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.418094 27343 raft_consensus.cc:357] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.418193 27343 raft_consensus.cc:383] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.418232 27343 raft_consensus.cc:738] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.418344 27350 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: No bootstrap required, opened a new log
I20250626 02:04:23.417069 27132 tablet_service.cc:1468] Processing CreateTablet for tablet 72a83458cfda49a6a0fbff57a1371c22 (DEFAULT_TABLE table=test-workload [id=7a13b057fc834f76b69fac55c008b9c6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:23.418344 27343 consensus_queue.cc:260] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.418447 27350 ts_tablet_manager.cc:1397] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:04:23.418563 27132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a83458cfda49a6a0fbff57a1371c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.418642 27343 ts_tablet_manager.cc:1428] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.418807 27343 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2: Bootstrap starting.
I20250626 02:04:23.418848 27350 raft_consensus.cc:357] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.418937 27350 raft_consensus.cc:383] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.418967 27350 raft_consensus.cc:738] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.419127 27350 consensus_queue.cc:260] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.419456 27350 ts_tablet_manager.cc:1428] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:23.419715 27343 tablet_bootstrap.cc:654] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.421499 27350 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:23.421574 27342 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41: No bootstrap required, opened a new log
I20250626 02:04:23.421736 27342 ts_tablet_manager.cc:1397] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250626 02:04:23.422082 27343 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2: No bootstrap required, opened a new log
I20250626 02:04:23.422139 27343 ts_tablet_manager.cc:1397] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250626 02:04:23.422206 27342 raft_consensus.cc:357] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.422307 27342 raft_consensus.cc:383] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.422351 27342 raft_consensus.cc:738] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.422416 27343 raft_consensus.cc:357] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.422516 27343 raft_consensus.cc:383] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.422542 27343 raft_consensus.cc:738] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.422495 27342 consensus_queue.cc:260] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.422624 27343 consensus_queue.cc:260] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.422787 27342 ts_tablet_manager.cc:1428] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.422873 27343 ts_tablet_manager.cc:1428] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.422962 27342 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41: Bootstrap starting.
I20250626 02:04:23.423906 27342 tablet_bootstrap.cc:654] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.425179 27342 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41: No bootstrap required, opened a new log
I20250626 02:04:23.425268 27342 ts_tablet_manager.cc:1397] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:23.425433 27350 tablet_bootstrap.cc:654] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.425649 27342 raft_consensus.cc:357] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.425769 27342 raft_consensus.cc:383] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.425824 27342 raft_consensus.cc:738] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.425937 27342 consensus_queue.cc:260] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.426185 27342 ts_tablet_manager.cc:1428] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:23.426667 27350 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: No bootstrap required, opened a new log
I20250626 02:04:23.426759 27350 ts_tablet_manager.cc:1397] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 02:04:23.427196 27350 raft_consensus.cc:357] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.427312 27350 raft_consensus.cc:383] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.427359 27350 raft_consensus.cc:738] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.427460 27350 consensus_queue.cc:260] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.427728 27350 ts_tablet_manager.cc:1428] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:23.427935 27350 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:23.428745 27350 tablet_bootstrap.cc:654] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.429761 27350 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: No bootstrap required, opened a new log
I20250626 02:04:23.429908 27350 ts_tablet_manager.cc:1397] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:23.430338 27350 raft_consensus.cc:357] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.430419 27350 raft_consensus.cc:383] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.430461 27350 raft_consensus.cc:738] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.430552 27350 consensus_queue.cc:260] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.430802 27350 ts_tablet_manager.cc:1428] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.430974 27350 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:23.431887 27350 tablet_bootstrap.cc:654] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.432972 27350 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: No bootstrap required, opened a new log
I20250626 02:04:23.433053 27350 ts_tablet_manager.cc:1397] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:04:23.433387 27350 raft_consensus.cc:357] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.433478 27350 raft_consensus.cc:383] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.433514 27350 raft_consensus.cc:738] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.433632 27350 consensus_queue.cc:260] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.433858 27346 raft_consensus.cc:491] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:23.433883 27350 ts_tablet_manager.cc:1428] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.433956 27346 raft_consensus.cc:513] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.434355 27346 leader_election.cc:290] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:23.435915 27346 raft_consensus.cc:491] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:23.436003 27346 raft_consensus.cc:513] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.436358 27346 leader_election.cc:290] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:23.438431 27296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf3aa09af90400190885a70f7c47362" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41" is_pre_election: true
I20250626 02:04:23.438508 27145 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf3aa09af90400190885a70f7c47362" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b" is_pre_election: true
I20250626 02:04:23.438551 27296 raft_consensus.cc:2466] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 0.
I20250626 02:04:23.438521 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72a83458cfda49a6a0fbff57a1371c22" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b" is_pre_election: true
I20250626 02:04:23.438746 27144 raft_consensus.cc:2466] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 0.
I20250626 02:04:23.438792 27295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72a83458cfda49a6a0fbff57a1371c22" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41" is_pre_election: true
I20250626 02:04:23.438650 27145 raft_consensus.cc:2466] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 0.
I20250626 02:04:23.438866 27295 raft_consensus.cc:2466] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 0.
I20250626 02:04:23.438881 27187 leader_election.cc:304] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822f01a34c91459993dd88632e7cfe41, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.439075 27187 leader_election.cc:304] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822f01a34c91459993dd88632e7cfe41, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.439360 27362 raft_consensus.cc:2802] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:23.439365 27346 raft_consensus.cc:2802] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:23.439461 27362 raft_consensus.cc:491] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:23.439469 27346 raft_consensus.cc:491] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:23.439545 27362 raft_consensus.cc:3058] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.439549 27346 raft_consensus.cc:3058] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.440572 27362 raft_consensus.cc:513] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.440632 27346 raft_consensus.cc:513] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.440912 27346 leader_election.cc:290] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:23.440913 27362 leader_election.cc:290] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:23.441264 27145 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf3aa09af90400190885a70f7c47362" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b"
I20250626 02:04:23.441327 27295 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72a83458cfda49a6a0fbff57a1371c22" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41"
I20250626 02:04:23.441381 27145 raft_consensus.cc:3058] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.441383 27296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf3aa09af90400190885a70f7c47362" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41"
I20250626 02:04:23.441519 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72a83458cfda49a6a0fbff57a1371c22" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b"
I20250626 02:04:23.441443 27295 raft_consensus.cc:3058] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.441612 27144 raft_consensus.cc:3058] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.442252 27145 raft_consensus.cc:2466] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 1.
I20250626 02:04:23.441537 27296 raft_consensus.cc:3058] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.442516 27184 leader_election.cc:304] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.442546 27144 raft_consensus.cc:2466] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 1.
I20250626 02:04:23.442546 27295 raft_consensus.cc:2466] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 1.
I20250626 02:04:23.442700 27362 raft_consensus.cc:2802] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:23.442802 27362 raft_consensus.cc:695] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Becoming Leader. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Running, Role: LEADER
I20250626 02:04:23.442843 27184 leader_election.cc:304] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.442929 27362 consensus_queue.cc:237] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.443256 27346 raft_consensus.cc:2802] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:23.443445 27346 raft_consensus.cc:695] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Becoming Leader. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Running, Role: LEADER
I20250626 02:04:23.443589 27346 consensus_queue.cc:237] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.443764 27296 raft_consensus.cc:2466] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 1.
I20250626 02:04:23.444453 27042 catalog_manager.cc:5582] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:23.446985 27042 catalog_manager.cc:5582] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:23.630903 27346 raft_consensus.cc:491] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:23.631053 27346 raft_consensus.cc:513] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.631491 27346 leader_election.cc:290] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677), 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353)
I20250626 02:04:23.631951 27296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06faf0dbd13448d4a2bf7bb4edfce338" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41" is_pre_election: true
I20250626 02:04:23.631951 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06faf0dbd13448d4a2bf7bb4edfce338" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b" is_pre_election: true
I20250626 02:04:23.632158 27144 raft_consensus.cc:2466] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 0.
I20250626 02:04:23.632172 27296 raft_consensus.cc:2466] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 0.
I20250626 02:04:23.632464 27187 leader_election.cc:304] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822f01a34c91459993dd88632e7cfe41, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.632632 27346 raft_consensus.cc:2802] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:23.632689 27346 raft_consensus.cc:491] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:23.632738 27346 raft_consensus.cc:3058] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.633654 27346 raft_consensus.cc:513] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.633952 27346 leader_election.cc:290] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 election: Requested vote from peers 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677), 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353)
I20250626 02:04:23.634256 27296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06faf0dbd13448d4a2bf7bb4edfce338" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41"
I20250626 02:04:23.634256 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06faf0dbd13448d4a2bf7bb4edfce338" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b"
I20250626 02:04:23.634392 27296 raft_consensus.cc:3058] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.634392 27144 raft_consensus.cc:3058] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.635368 27296 raft_consensus.cc:2466] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 1.
I20250626 02:04:23.635368 27144 raft_consensus.cc:2466] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 1.
I20250626 02:04:23.635656 27184 leader_election.cc:304] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.635802 27346 raft_consensus.cc:2802] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:23.635890 27346 raft_consensus.cc:695] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Becoming Leader. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Running, Role: LEADER
I20250626 02:04:23.636039 27346 consensus_queue.cc:237] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:23.637588 27042 catalog_manager.cc:5582] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:23.692786 27348 raft_consensus.cc:491] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:23.692979 27348 raft_consensus.cc:513] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.693516 27348 leader_election.cc:290] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:23.697979 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef6548b73bbc43229fc8f974b4165ac7" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b" is_pre_election: true
I20250626 02:04:23.697988 27221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef6548b73bbc43229fc8f974b4165ac7" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" is_pre_election: true
I20250626 02:04:23.698130 27144 raft_consensus.cc:2466] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 0.
I20250626 02:04:23.698225 27221 raft_consensus.cc:2466] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 0.
I20250626 02:04:23.698446 27259 leader_election.cc:304] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, 822f01a34c91459993dd88632e7cfe41; no voters:
I20250626 02:04:23.698660 27348 raft_consensus.cc:2802] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:23.698709 27348 raft_consensus.cc:491] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:23.698745 27348 raft_consensus.cc:3058] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.699553 27348 raft_consensus.cc:513] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.699842 27348 leader_election.cc:290] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 election: Requested vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:23.700189 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef6548b73bbc43229fc8f974b4165ac7" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b"
I20250626 02:04:23.700222 27221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef6548b73bbc43229fc8f974b4165ac7" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26ebacebabe43d6822b5a63ff17c6e2"
I20250626 02:04:23.700276 27144 raft_consensus.cc:3058] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.700325 27221 raft_consensus.cc:3058] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.701021 27144 raft_consensus.cc:2466] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 1.
I20250626 02:04:23.701120 27221 raft_consensus.cc:2466] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 1.
I20250626 02:04:23.701289 27259 leader_election.cc:304] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, 822f01a34c91459993dd88632e7cfe41; no voters:
I20250626 02:04:23.701505 27348 raft_consensus.cc:2802] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:23.701689 27348 raft_consensus.cc:695] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Becoming Leader. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Running, Role: LEADER
I20250626 02:04:23.701836 27348 consensus_queue.cc:237] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.703284 27042 catalog_manager.cc:5582] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 822f01a34c91459993dd88632e7cfe41 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "822f01a34c91459993dd88632e7cfe41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:23.766263 20338 test_util.cc:276] Using random seed: -83990293
I20250626 02:04:23.774103 27042 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38534:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:04:23.774633 27042 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:04:23.780222 27209 tablet_service.cc:1468] Processing CreateTablet for tablet fc88aaa9948348f5b4cb8fb2c31a61c7 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:23.780390 27132 tablet_service.cc:1468] Processing CreateTablet for tablet d904a02cce9348ffae84798371b0f5ca (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:23.780519 27209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc88aaa9948348f5b4cb8fb2c31a61c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.780637 27132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d904a02cce9348ffae84798371b0f5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.781000 27285 tablet_service.cc:1468] Processing CreateTablet for tablet e7ba4f3151ce49fe8524ae6834dc64f1 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:23.781144 27133 tablet_service.cc:1468] Processing CreateTablet for tablet fc88aaa9948348f5b4cb8fb2c31a61c7 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:23.781241 27285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7ba4f3151ce49fe8524ae6834dc64f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.781342 27133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc88aaa9948348f5b4cb8fb2c31a61c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.782827 27343 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: Bootstrap starting.
I20250626 02:04:23.783407 27342 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: Bootstrap starting.
I20250626 02:04:23.783527 27284 tablet_service.cc:1468] Processing CreateTablet for tablet fc88aaa9948348f5b4cb8fb2c31a61c7 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 02:04:23.783596 27343 tablet_bootstrap.cc:654] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.783741 27284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc88aaa9948348f5b4cb8fb2c31a61c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.784288 27350 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:23.784370 27342 tablet_bootstrap.cc:654] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.784994 27350 tablet_bootstrap.cc:654] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.785681 27283 tablet_service.cc:1468] Processing CreateTablet for tablet d904a02cce9348ffae84798371b0f5ca (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:23.785791 27342 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: No bootstrap required, opened a new log
I20250626 02:04:23.785866 27342 ts_tablet_manager.cc:1397] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:23.785899 27283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d904a02cce9348ffae84798371b0f5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.786224 27350 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: No bootstrap required, opened a new log
I20250626 02:04:23.786286 27350 ts_tablet_manager.cc:1397] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250626 02:04:23.786279 27342 raft_consensus.cc:357] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.786394 27342 raft_consensus.cc:383] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.786427 27342 raft_consensus.cc:738] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.786546 27342 consensus_queue.cc:260] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.786655 27350 raft_consensus.cc:357] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.786760 27350 raft_consensus.cc:383] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.786795 27350 raft_consensus.cc:738] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.786819 27342 ts_tablet_manager.cc:1428] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.786895 27350 consensus_queue.cc:260] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.787011 27342 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: Bootstrap starting.
I20250626 02:04:23.787140 27350 ts_tablet_manager.cc:1428] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:23.787299 27350 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:23.787971 27342 tablet_bootstrap.cc:654] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.788067 27350 tablet_bootstrap.cc:654] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.788492 27133 tablet_service.cc:1468] Processing CreateTablet for tablet e7ba4f3151ce49fe8524ae6834dc64f1 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:23.788751 27133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7ba4f3151ce49fe8524ae6834dc64f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.789219 27350 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: No bootstrap required, opened a new log
I20250626 02:04:23.789325 27350 ts_tablet_manager.cc:1397] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:23.789757 27350 raft_consensus.cc:357] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.789858 27350 raft_consensus.cc:383] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.789894 27350 raft_consensus.cc:738] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.790025 27350 consensus_queue.cc:260] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.790287 27350 ts_tablet_manager.cc:1428] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.790407 27132 tablet_service.cc:1468] Processing CreateTablet for tablet 4212c38370ab4dfe9720fd1f788bb791 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:23.790640 27132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4212c38370ab4dfe9720fd1f788bb791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.789234 27342 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: No bootstrap required, opened a new log
I20250626 02:04:23.791070 27342 ts_tablet_manager.cc:1397] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250626 02:04:23.791332 27211 tablet_service.cc:1468] Processing CreateTablet for tablet e7ba4f3151ce49fe8524ae6834dc64f1 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 02:04:23.791465 27342 raft_consensus.cc:357] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.791553 27342 raft_consensus.cc:383] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.791567 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7ba4f3151ce49fe8524ae6834dc64f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.791587 27342 raft_consensus.cc:738] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.791700 27342 consensus_queue.cc:260] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.791939 27342 ts_tablet_manager.cc:1428] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.792083 27342 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: Bootstrap starting.
I20250626 02:04:23.792326 27209 tablet_service.cc:1468] Processing CreateTablet for tablet d904a02cce9348ffae84798371b0f5ca (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 02:04:23.792379 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 4212c38370ab4dfe9720fd1f788bb791 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:23.792557 27209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d904a02cce9348ffae84798371b0f5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.792809 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4212c38370ab4dfe9720fd1f788bb791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.793792 27283 tablet_service.cc:1468] Processing CreateTablet for tablet 4212c38370ab4dfe9720fd1f788bb791 (DEFAULT_TABLE table=dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 02:04:23.794191 27283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4212c38370ab4dfe9720fd1f788bb791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:23.794277 27350 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:23.795106 27350 tablet_bootstrap.cc:654] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.796082 27350 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: No bootstrap required, opened a new log
I20250626 02:04:23.796191 27350 ts_tablet_manager.cc:1397] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:04:23.796602 27350 raft_consensus.cc:357] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.796726 27350 raft_consensus.cc:383] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.796761 27350 raft_consensus.cc:738] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.797220 27350 consensus_queue.cc:260] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.797593 27350 ts_tablet_manager.cc:1428] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:23.797765 27350 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:23.797974 27342 tablet_bootstrap.cc:654] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.798782 27350 tablet_bootstrap.cc:654] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.799098 27342 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: No bootstrap required, opened a new log
I20250626 02:04:23.799202 27342 ts_tablet_manager.cc:1397] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250626 02:04:23.799618 27342 raft_consensus.cc:357] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.799701 27342 raft_consensus.cc:383] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.799731 27342 raft_consensus.cc:738] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.799728 27343 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: No bootstrap required, opened a new log
I20250626 02:04:23.799852 27343 ts_tablet_manager.cc:1397] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250626 02:04:23.800166 27350 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: No bootstrap required, opened a new log
I20250626 02:04:23.800271 27350 ts_tablet_manager.cc:1397] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:23.800267 27343 raft_consensus.cc:357] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.800415 27343 raft_consensus.cc:383] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.800464 27343 raft_consensus.cc:738] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.800587 27343 consensus_queue.cc:260] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.800671 27350 raft_consensus.cc:357] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.800766 27350 raft_consensus.cc:383] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.800804 27350 raft_consensus.cc:738] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.800866 27343 ts_tablet_manager.cc:1428] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:23.800904 27350 consensus_queue.cc:260] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.801038 27342 consensus_queue.cc:260] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.801174 27350 ts_tablet_manager.cc:1428] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.801400 27342 ts_tablet_manager.cc:1428] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 02:04:23.801586 27342 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: Bootstrap starting.
I20250626 02:04:23.801100 27343 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: Bootstrap starting.
I20250626 02:04:23.802590 27342 tablet_bootstrap.cc:654] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.802632 27343 tablet_bootstrap.cc:654] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.803660 27343 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: No bootstrap required, opened a new log
I20250626 02:04:23.803764 27343 ts_tablet_manager.cc:1397] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250626 02:04:23.804445 27343 raft_consensus.cc:357] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.804575 27343 raft_consensus.cc:383] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.804620 27343 raft_consensus.cc:738] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.804767 27343 consensus_queue.cc:260] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.805012 27343 ts_tablet_manager.cc:1428] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:23.805174 27343 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: Bootstrap starting.
I20250626 02:04:23.805891 27343 tablet_bootstrap.cc:654] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.804459 27342 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: No bootstrap required, opened a new log
I20250626 02:04:23.806416 27342 ts_tablet_manager.cc:1397] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20250626 02:04:23.806761 27342 raft_consensus.cc:357] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.806788 27343 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: No bootstrap required, opened a new log
I20250626 02:04:23.806905 27342 raft_consensus.cc:383] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.806948 27343 ts_tablet_manager.cc:1397] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:23.806953 27342 raft_consensus.cc:738] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.807117 27342 consensus_queue.cc:260] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.807353 27342 ts_tablet_manager.cc:1428] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:23.807384 27343 raft_consensus.cc:357] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.807499 27343 raft_consensus.cc:383] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.807543 27343 raft_consensus.cc:738] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.807655 27343 consensus_queue.cc:260] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.807917 27343 ts_tablet_manager.cc:1428] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:23.808115 27343 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: Bootstrap starting.
I20250626 02:04:23.809273 27343 tablet_bootstrap.cc:654] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: Neither blocks nor log segments found. Creating new log.
I20250626 02:04:23.810348 27343 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: No bootstrap required, opened a new log
I20250626 02:04:23.810452 27343 ts_tablet_manager.cc:1397] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250626 02:04:23.810801 27343 raft_consensus.cc:357] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.810894 27343 raft_consensus.cc:383] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:04:23.810930 27343 raft_consensus.cc:738] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Initialized, Role: FOLLOWER
I20250626 02:04:23.811055 27343 consensus_queue.cc:260] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.811316 27343 ts_tablet_manager.cc:1428] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:23.815224 27348 raft_consensus.cc:491] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:23.815307 27348 raft_consensus.cc:513] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.815594 27348 leader_election.cc:290] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:23.815910 27221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4212c38370ab4dfe9720fd1f788bb791" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" is_pre_election: true
I20250626 02:04:23.815999 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4212c38370ab4dfe9720fd1f788bb791" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b" is_pre_election: true
I20250626 02:04:23.816016 27221 raft_consensus.cc:2466] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 0.
I20250626 02:04:23.816162 27144 raft_consensus.cc:2466] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 0.
I20250626 02:04:23.816408 27260 leader_election.cc:304] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822f01a34c91459993dd88632e7cfe41, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.816584 27348 raft_consensus.cc:2802] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:23.816654 27348 raft_consensus.cc:491] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:23.816721 27348 raft_consensus.cc:3058] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.817519 27348 raft_consensus.cc:513] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.817849 27348 leader_election.cc:290] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 election: Requested vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:23.818097 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4212c38370ab4dfe9720fd1f788bb791" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b"
I20250626 02:04:23.818122 27221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4212c38370ab4dfe9720fd1f788bb791" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26ebacebabe43d6822b5a63ff17c6e2"
I20250626 02:04:23.818218 27144 raft_consensus.cc:3058] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.818257 27221 raft_consensus.cc:3058] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.818965 27144 raft_consensus.cc:2466] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 1.
I20250626 02:04:23.819028 27221 raft_consensus.cc:2466] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 1.
I20250626 02:04:23.819217 27259 leader_election.cc:304] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, 822f01a34c91459993dd88632e7cfe41; no voters:
I20250626 02:04:23.819427 27348 raft_consensus.cc:2802] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:23.819517 27348 raft_consensus.cc:695] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Becoming Leader. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Running, Role: LEADER
I20250626 02:04:23.819643 27348 consensus_queue.cc:237] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.821120 27042 catalog_manager.cc:5582] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 822f01a34c91459993dd88632e7cfe41 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "822f01a34c91459993dd88632e7cfe41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:23.823180 27348 raft_consensus.cc:491] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:23.823271 27348 raft_consensus.cc:513] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.823510 27348 leader_election.cc:290] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:23.823776 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc88aaa9948348f5b4cb8fb2c31a61c7" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b" is_pre_election: true
I20250626 02:04:23.823836 27221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc88aaa9948348f5b4cb8fb2c31a61c7" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" is_pre_election: true
I20250626 02:04:23.823886 27144 raft_consensus.cc:2466] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 0.
I20250626 02:04:23.823947 27221 raft_consensus.cc:2466] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 0.
I20250626 02:04:23.824186 27260 leader_election.cc:304] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822f01a34c91459993dd88632e7cfe41, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.824379 27348 raft_consensus.cc:2802] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:23.824429 27348 raft_consensus.cc:491] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:23.824494 27348 raft_consensus.cc:3058] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.825184 27348 raft_consensus.cc:513] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.825469 27348 leader_election.cc:290] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 election: Requested vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:23.825750 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc88aaa9948348f5b4cb8fb2c31a61c7" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b"
I20250626 02:04:23.825767 27221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc88aaa9948348f5b4cb8fb2c31a61c7" candidate_uuid: "822f01a34c91459993dd88632e7cfe41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26ebacebabe43d6822b5a63ff17c6e2"
I20250626 02:04:23.825834 27144 raft_consensus.cc:3058] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.825866 27221 raft_consensus.cc:3058] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.826535 27221 raft_consensus.cc:2466] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 1.
I20250626 02:04:23.826534 27144 raft_consensus.cc:2466] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 822f01a34c91459993dd88632e7cfe41 in term 1.
I20250626 02:04:23.826886 27259 leader_election.cc:304] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, 822f01a34c91459993dd88632e7cfe41; no voters:
I20250626 02:04:23.827028 27348 raft_consensus.cc:2802] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:23.827095 27348 raft_consensus.cc:695] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Becoming Leader. State: Replica: 822f01a34c91459993dd88632e7cfe41, State: Running, Role: LEADER
I20250626 02:04:23.827208 27348 consensus_queue.cc:237] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.828694 27042 catalog_manager.cc:5582] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 822f01a34c91459993dd88632e7cfe41 (127.19.220.131). New cstate: current_term: 1 leader_uuid: "822f01a34c91459993dd88632e7cfe41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:23.845178 27354 raft_consensus.cc:491] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:23.845319 27354 raft_consensus.cc:513] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.845834 27354 leader_election.cc:290] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589), 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:23.850080 27296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d904a02cce9348ffae84798371b0f5ca" candidate_uuid: "7b20a67f201e4924a8f003667eeb604b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41" is_pre_election: true
I20250626 02:04:23.850090 27221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d904a02cce9348ffae84798371b0f5ca" candidate_uuid: "7b20a67f201e4924a8f003667eeb604b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" is_pre_election: true
I20250626 02:04:23.850211 27296 raft_consensus.cc:2466] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b20a67f201e4924a8f003667eeb604b in term 0.
I20250626 02:04:23.850255 27221 raft_consensus.cc:2466] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b20a67f201e4924a8f003667eeb604b in term 0.
I20250626 02:04:23.850489 27111 leader_election.cc:304] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, 822f01a34c91459993dd88632e7cfe41; no voters:
I20250626 02:04:23.850705 27354 raft_consensus.cc:2802] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:23.850764 27354 raft_consensus.cc:491] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:23.850792 27354 raft_consensus.cc:3058] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.851567 27354 raft_consensus.cc:513] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.851895 27354 leader_election.cc:290] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [CANDIDATE]: Term 1 election: Requested vote from peers d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589), 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:23.852209 27221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d904a02cce9348ffae84798371b0f5ca" candidate_uuid: "7b20a67f201e4924a8f003667eeb604b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26ebacebabe43d6822b5a63ff17c6e2"
I20250626 02:04:23.852221 27296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d904a02cce9348ffae84798371b0f5ca" candidate_uuid: "7b20a67f201e4924a8f003667eeb604b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41"
I20250626 02:04:23.852340 27221 raft_consensus.cc:3058] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.852340 27296 raft_consensus.cc:3058] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.853294 27221 raft_consensus.cc:2466] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b20a67f201e4924a8f003667eeb604b in term 1.
I20250626 02:04:23.853294 27296 raft_consensus.cc:2466] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b20a67f201e4924a8f003667eeb604b in term 1.
I20250626 02:04:23.853621 27109 leader_election.cc:304] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b20a67f201e4924a8f003667eeb604b, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.853782 27354 raft_consensus.cc:2802] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:23.853996 27354 raft_consensus.cc:695] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 LEADER]: Becoming Leader. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Running, Role: LEADER
I20250626 02:04:23.854135 27354 consensus_queue.cc:237] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.855630 27042 catalog_manager.cc:5582] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b20a67f201e4924a8f003667eeb604b (127.19.220.129). New cstate: current_term: 1 leader_uuid: "7b20a67f201e4924a8f003667eeb604b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:23.896255 27346 raft_consensus.cc:491] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:04:23.896400 27346 raft_consensus.cc:513] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.896804 27346 leader_election.cc:290] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:23.897223 27296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7ba4f3151ce49fe8524ae6834dc64f1" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41" is_pre_election: true
I20250626 02:04:23.897219 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7ba4f3151ce49fe8524ae6834dc64f1" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b" is_pre_election: true
I20250626 02:04:23.897373 27296 raft_consensus.cc:2466] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 0.
I20250626 02:04:23.897393 27144 raft_consensus.cc:2466] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 0.
I20250626 02:04:23.897648 27187 leader_election.cc:304] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822f01a34c91459993dd88632e7cfe41, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.897817 27346 raft_consensus.cc:2802] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:04:23.897872 27346 raft_consensus.cc:491] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:04:23.897928 27346 raft_consensus.cc:3058] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.898790 27346 raft_consensus.cc:513] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.899108 27346 leader_election.cc:290] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 election: Requested vote from peers 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353), 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:23.899377 27144 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7ba4f3151ce49fe8524ae6834dc64f1" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b20a67f201e4924a8f003667eeb604b"
I20250626 02:04:23.899413 27296 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7ba4f3151ce49fe8524ae6834dc64f1" candidate_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822f01a34c91459993dd88632e7cfe41"
I20250626 02:04:23.899475 27144 raft_consensus.cc:3058] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.899504 27296 raft_consensus.cc:3058] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:04:23.900288 27296 raft_consensus.cc:2466] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 1.
I20250626 02:04:23.900404 27144 raft_consensus.cc:2466] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26ebacebabe43d6822b5a63ff17c6e2 in term 1.
I20250626 02:04:23.900552 27187 leader_election.cc:304] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822f01a34c91459993dd88632e7cfe41, d26ebacebabe43d6822b5a63ff17c6e2; no voters:
I20250626 02:04:23.900759 27346 raft_consensus.cc:2802] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:04:23.900851 27346 raft_consensus.cc:695] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Becoming Leader. State: Replica: d26ebacebabe43d6822b5a63ff17c6e2, State: Running, Role: LEADER
I20250626 02:04:23.900982 27346 consensus_queue.cc:237] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:23.902419 27042 catalog_manager.cc:5582] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130). New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:23.911568 20338 tablet_server.cc:178] TabletServer@127.19.220.129:0 shutting down...
I20250626 02:04:23.915221 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:23.915477 20338 tablet_replica.cc:331] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: stopping tablet replica
I20250626 02:04:23.915593 20338 raft_consensus.cc:2241] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:23.915690 20338 raft_consensus.cc:2270] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:23.916162 20338 tablet_replica.cc:331] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: stopping tablet replica
I20250626 02:04:23.916271 20338 raft_consensus.cc:2241] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:23.916360 20338 raft_consensus.cc:2270] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:23.916750 20338 tablet_replica.cc:331] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: stopping tablet replica
I20250626 02:04:23.916823 20338 raft_consensus.cc:2241] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:23.916891 20338 raft_consensus.cc:2270] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:23.917238 20338 tablet_replica.cc:331] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: stopping tablet replica
I20250626 02:04:23.917312 20338 raft_consensus.cc:2241] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:23.917378 20338 raft_consensus.cc:2270] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:23.917673 20338 tablet_replica.cc:331] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: stopping tablet replica
I20250626 02:04:23.917750 20338 raft_consensus.cc:2241] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:23.917856 20338 pending_rounds.cc:70] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Trying to abort 1 pending ops.
I20250626 02:04:23.917881 20338 pending_rounds.cc:77] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171700587947921408 op_type: NO_OP noop_request { }
I20250626 02:04:23.917938 20338 raft_consensus.cc:2887] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:04:23.917977 20338 raft_consensus.cc:2270] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:23.918434 20338 tablet_replica.cc:331] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: stopping tablet replica
I20250626 02:04:23.918514 20338 raft_consensus.cc:2241] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:23.918581 20338 raft_consensus.cc:2270] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:23.918938 20338 tablet_replica.cc:331] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: stopping tablet replica
I20250626 02:04:23.919009 20338 raft_consensus.cc:2241] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:23.919077 20338 raft_consensus.cc:2270] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:23.919309 20338 tablet_replica.cc:331] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: stopping tablet replica
I20250626 02:04:23.919373 20338 raft_consensus.cc:2241] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:23.919435 20338 raft_consensus.cc:2270] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 02:04:23.930011 27184 consensus_peers.cc:487] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): Couldn't send request to peer 7b20a67f201e4924a8f003667eeb604b. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:37353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 02:04:23.930572 27184 consensus_peers.cc:487] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): Couldn't send request to peer 7b20a67f201e4924a8f003667eeb604b. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:37353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:23.933135 20338 tablet_server.cc:195] TabletServer@127.19.220.129:0 shutdown complete.
I20250626 02:04:24.033279 27346 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:24.043000 27346 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 02:04:24.138934 27259 consensus_peers.cc:487] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 -> Peer 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): Couldn't send request to peer 7b20a67f201e4924a8f003667eeb604b. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:37353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 02:04:24.144122 27184 consensus_peers.cc:487] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): Couldn't send request to peer 7b20a67f201e4924a8f003667eeb604b. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:37353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:24.240386 27348 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:24.257715 27362 consensus_queue.cc:1035] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 02:04:24.268952 27259 consensus_peers.cc:487] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 -> Peer 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): Couldn't send request to peer 7b20a67f201e4924a8f003667eeb604b. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:37353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 02:04:24.294278 27259 consensus_peers.cc:487] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 -> Peer 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): Couldn't send request to peer 7b20a67f201e4924a8f003667eeb604b. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:37353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:24.366433 27348 consensus_queue.cc:1035] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:24.438561 27348 consensus_queue.cc:1035] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 02:04:24.443598 27184 consensus_peers.cc:487] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): Couldn't send request to peer 7b20a67f201e4924a8f003667eeb604b. Status: Network error: Client connection negotiation failed: client connection to 127.19.220.129:37353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:24.455143 27362 consensus_queue.cc:1035] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:24.486194 27362 consensus_queue.cc:579] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Leader has been unable to successfully communicate with peer 7b20a67f201e4924a8f003667eeb604b for more than 1 seconds (1.042s)
I20250626 02:04:24.723892 27406 consensus_queue.cc:579] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Leader has been unable to successfully communicate with peer 7b20a67f201e4924a8f003667eeb604b for more than 1 seconds (1.087s)
I20250626 02:04:24.940706 27040 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38530:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250626 02:04:24.940853 27040 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38530:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250626 02:04:24.944016 27040 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: Sending DeleteTablet for 3 replicas of tablet fc88aaa9948348f5b4cb8fb2c31a61c7
I20250626 02:04:24.944382 27040 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: Sending DeleteTablet for 3 replicas of tablet d904a02cce9348ffae84798371b0f5ca
I20250626 02:04:24.944612 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet fc88aaa9948348f5b4cb8fb2c31a61c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:53184
I20250626 02:04:24.944666 27040 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: Sending DeleteTablet for 3 replicas of tablet e7ba4f3151ce49fe8524ae6834dc64f1
I20250626 02:04:24.944689 27283 tablet_service.cc:1515] Processing DeleteTablet for tablet fc88aaa9948348f5b4cb8fb2c31a61c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40862
I20250626 02:04:24.944880 27040 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: Sending DeleteTablet for 3 replicas of tablet 4212c38370ab4dfe9720fd1f788bb791
I20250626 02:04:24.944981 27415 tablet_replica.cc:331] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: stopping tablet replica
I20250626 02:04:24.945112 27415 raft_consensus.cc:2241] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:24.945184 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet d904a02cce9348ffae84798371b0f5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:53184
I20250626 02:04:24.945268 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet e7ba4f3151ce49fe8524ae6834dc64f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:53184
I20250626 02:04:24.945322 27415 raft_consensus.cc:2270] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:24.945364 27414 tablet_replica.cc:331] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: stopping tablet replica
I20250626 02:04:24.945389 27284 tablet_service.cc:1515] Processing DeleteTablet for tablet e7ba4f3151ce49fe8524ae6834dc64f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40862
I20250626 02:04:24.945487 27414 raft_consensus.cc:2241] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:24.945519 27283 tablet_service.cc:1515] Processing DeleteTablet for tablet d904a02cce9348ffae84798371b0f5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40862
I20250626 02:04:24.945600 27414 raft_consensus.cc:2270] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:24.945832 27283 tablet_service.cc:1515] Processing DeleteTablet for tablet 4212c38370ab4dfe9720fd1f788bb791 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40862
I20250626 02:04:24.945966 27415 ts_tablet_manager.cc:1905] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:24.946177 27414 ts_tablet_manager.cc:1905] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:24.946346 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 02:04:24.947023 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 4212c38370ab4dfe9720fd1f788bb791 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:53184
W20250626 02:04:24.947206 27026 catalog_manager.cc:4671] TS 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d904a02cce9348ffae84798371b0f5ca: Network error: Client connection negotiation failed: client connection to 127.19.220.129:37353: connect: Connection refused (error 111)
W20250626 02:04:24.948168 27420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:24.948573 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:24.948601 27418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:24.948846 27415 ts_tablet_manager.cc:1918] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 02:04:24.948928 27415 log.cc:1199] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/fc88aaa9948348f5b4cb8fb2c31a61c7
I20250626 02:04:24.948935 27414 ts_tablet_manager.cc:1918] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250626 02:04:24.949080 27417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:24.949111 27414 log.cc:1199] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/fc88aaa9948348f5b4cb8fb2c31a61c7
I20250626 02:04:24.949249 27415 ts_tablet_manager.cc:1939] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 822f01a34c91459993dd88632e7cfe41: Deleting consensus metadata
I20250626 02:04:24.949363 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:24.949424 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:24.949471 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903464949470 us; error 0 us; skew 500 ppm
I20250626 02:04:24.949429 27414 ts_tablet_manager.cc:1939] T fc88aaa9948348f5b4cb8fb2c31a61c7 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting consensus metadata
I20250626 02:04:24.949889 27415 tablet_replica.cc:331] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: stopping tablet replica
I20250626 02:04:24.950052 27415 raft_consensus.cc:2241] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:24.950088 27414 tablet_replica.cc:331] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: stopping tablet replica
I20250626 02:04:24.950156 27415 raft_consensus.cc:2270] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:24.950171 27027 catalog_manager.cc:4928] TS d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589): tablet fc88aaa9948348f5b4cb8fb2c31a61c7 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:24.950206 27414 raft_consensus.cc:2241] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:24.950287 27414 raft_consensus.cc:2270] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:24.950331 27029 catalog_manager.cc:4928] TS 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677): tablet fc88aaa9948348f5b4cb8fb2c31a61c7 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:24.950681 27415 ts_tablet_manager.cc:1905] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:24.950685 27414 ts_tablet_manager.cc:1905] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:24.950776 20338 webserver.cc:469] Webserver started at http://127.19.220.132:37011/ using document root <none> and password file <none>
I20250626 02:04:24.950891 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:24.950922 20338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:04:24.950968 20338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:04:24.951227 20338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-3-root/instance:
uuid: "ecaca65e824c4aa48c4a06ec56b90eb6"
format_stamp: "Formatted at 2025-06-26 02:04:24 on dist-test-slave-h18f"
I20250626 02:04:24.952526 27415 ts_tablet_manager.cc:1918] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 02:04:24.952526 27414 ts_tablet_manager.cc:1918] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 02:04:24.952620 20338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250626 02:04:24.952644 27414 log.cc:1199] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/d904a02cce9348ffae84798371b0f5ca
I20250626 02:04:24.952638 27415 log.cc:1199] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/e7ba4f3151ce49fe8524ae6834dc64f1
I20250626 02:04:24.952884 27414 ts_tablet_manager.cc:1939] T d904a02cce9348ffae84798371b0f5ca P d26ebacebabe43d6822b5a63ff17c6e2: Deleting consensus metadata
I20250626 02:04:24.952915 27415 ts_tablet_manager.cc:1939] T e7ba4f3151ce49fe8524ae6834dc64f1 P 822f01a34c91459993dd88632e7cfe41: Deleting consensus metadata
I20250626 02:04:24.953428 27414 tablet_replica.cc:331] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: stopping tablet replica
I20250626 02:04:24.953497 27029 catalog_manager.cc:4928] TS 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677): tablet e7ba4f3151ce49fe8524ae6834dc64f1 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:24.953554 27414 raft_consensus.cc:2241] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:24.953606 27425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:24.953595 27027 catalog_manager.cc:4928] TS d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589): tablet d904a02cce9348ffae84798371b0f5ca (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:24.953671 27415 tablet_replica.cc:331] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: stopping tablet replica
I20250626 02:04:24.953717 27414 raft_consensus.cc:2270] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:24.953789 27415 raft_consensus.cc:2241] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:24.953878 27415 raft_consensus.cc:2270] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:24.954073 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:24.954140 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-3-root
uuid: "ecaca65e824c4aa48c4a06ec56b90eb6"
format_stamp: "Formatted at 2025-06-26 02:04:24 on dist-test-slave-h18f"
I20250626 02:04:24.954188 27414 ts_tablet_manager.cc:1905] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:24.954188 27415 ts_tablet_manager.cc:1905] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:24.954212 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:24.955973 27414 ts_tablet_manager.cc:1918] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 02:04:24.955973 27415 ts_tablet_manager.cc:1918] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 02:04:24.956079 27414 log.cc:1199] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/e7ba4f3151ce49fe8524ae6834dc64f1
I20250626 02:04:24.956080 27415 log.cc:1199] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/d904a02cce9348ffae84798371b0f5ca
I20250626 02:04:24.956349 27415 ts_tablet_manager.cc:1939] T d904a02cce9348ffae84798371b0f5ca P 822f01a34c91459993dd88632e7cfe41: Deleting consensus metadata
I20250626 02:04:24.956379 27414 ts_tablet_manager.cc:1939] T e7ba4f3151ce49fe8524ae6834dc64f1 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting consensus metadata
I20250626 02:04:24.957296 27414 tablet_replica.cc:331] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: stopping tablet replica
I20250626 02:04:24.957373 27027 catalog_manager.cc:4928] TS d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589): tablet e7ba4f3151ce49fe8524ae6834dc64f1 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:24.957438 27415 tablet_replica.cc:331] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: stopping tablet replica
I20250626 02:04:24.957422 27414 raft_consensus.cc:2241] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:24.957360 27029 catalog_manager.cc:4928] TS 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677): tablet d904a02cce9348ffae84798371b0f5ca (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:24.957531 27415 raft_consensus.cc:2241] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:24.957556 27414 raft_consensus.cc:2270] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:24.957697 27415 raft_consensus.cc:2270] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:24.958170 27415 ts_tablet_manager.cc:1905] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:24.958293 27414 ts_tablet_manager.cc:1905] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:24.959625 27415 ts_tablet_manager.cc:1918] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 02:04:24.959697 27415 log.cc:1199] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/4212c38370ab4dfe9720fd1f788bb791
I20250626 02:04:24.959924 27415 ts_tablet_manager.cc:1939] T 4212c38370ab4dfe9720fd1f788bb791 P 822f01a34c91459993dd88632e7cfe41: Deleting consensus metadata
I20250626 02:04:24.960000 27414 ts_tablet_manager.cc:1918] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 02:04:24.960075 27414 log.cc:1199] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/4212c38370ab4dfe9720fd1f788bb791
I20250626 02:04:24.960330 27414 ts_tablet_manager.cc:1939] T 4212c38370ab4dfe9720fd1f788bb791 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting consensus metadata
I20250626 02:04:24.960589 27029 catalog_manager.cc:4928] TS 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677): tablet 4212c38370ab4dfe9720fd1f788bb791 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:24.960914 27027 catalog_manager.cc:4928] TS d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589): tablet 4212c38370ab4dfe9720fd1f788bb791 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:24.962132 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:24.962411 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:24.962648 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:24.962990 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:04:24.963025 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:24.963054 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:04:24.963074 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:24.966954 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.132:33109
I20250626 02:04:24.967772 27488 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.132:33109 every 8 connection(s)
I20250626 02:04:24.967971 20338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:04:24.969914 27493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:04:24.970148 27495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:24.970199 20338 server_base.cc:1048] running on GCE node
W20250626 02:04:24.970479 27492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:04:24.970690 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 02:04:24.970731 20338 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 02:04:24.970755 20338 hybrid_clock.cc:648] HybridClock initialized: now 1750903464970755 us; error 0 us; skew 500 ppm
I20250626 02:04:24.971964 20338 webserver.cc:469] Webserver started at http://127.19.220.129:40835/ using document root <none> and password file <none>
I20250626 02:04:24.972122 20338 fs_manager.cc:362] Metadata directory not provided
I20250626 02:04:24.972191 20338 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250626 02:04:24.972306 27489 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42333
I20250626 02:04:24.972395 27489 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:24.972594 27489 heartbeater.cc:507] Master 127.19.220.190:42333 requested a full tablet report, sending...
I20250626 02:04:24.973147 27040 ts_manager.cc:194] Registered new tserver with Master: ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109)
I20250626 02:04:24.973275 20338 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:24.973685 27040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38554
I20250626 02:04:24.974558 27500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:04:24.974895 20338 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:24.974962 20338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root
uuid: "7b20a67f201e4924a8f003667eeb604b"
format_stamp: "Formatted at 2025-06-26 02:04:22 on dist-test-slave-h18f"
I20250626 02:04:24.975044 20338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:04:24.981884 20338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:04:24.982221 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:04:24.982497 20338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:04:24.983115 27507 ts_tablet_manager.cc:536] Loading tablet metadata (0/8 complete)
I20250626 02:04:24.988687 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20250626 02:04:24.988761 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20250626 02:04:24.988823 20338 ts_tablet_manager.cc:594] Registering tablets (0/8 complete)
I20250626 02:04:24.989562 27507 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:24.991657 27507 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:24.991888 27507 tablet_bootstrap.cc:492] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap complete.
I20250626 02:04:24.992054 27507 ts_tablet_manager.cc:1397] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:24.992424 27507 raft_consensus.cc:357] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:24.992529 27507 raft_consensus.cc:738] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:24.992681 20338 ts_tablet_manager.cc:610] Registered 8 tablets
I20250626 02:04:24.992748 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.003s
I20250626 02:04:24.992692 27507 consensus_queue.cc:260] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:24.993112 27507 ts_tablet_manager.cc:1428] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:24.993268 27507 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:24.995294 27507 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:24.995528 27507 tablet_bootstrap.cc:492] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap complete.
I20250626 02:04:24.995697 27507 ts_tablet_manager.cc:1397] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250626 02:04:24.996039 27507 raft_consensus.cc:357] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:24.996189 27507 raft_consensus.cc:738] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:24.996377 27507 consensus_queue.cc:260] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:24.996639 27507 ts_tablet_manager.cc:1428] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:24.996804 27507 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:24.998298 20338 rpc_server.cc:307] RPC server started. Bound to: 127.19.220.129:37353
I20250626 02:04:24.998344 27567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.220.129:37353 every 8 connection(s)
I20250626 02:04:24.998978 27507 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:24.999219 27507 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap complete.
I20250626 02:04:24.999397 27507 ts_tablet_manager.cc:1397] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250626 02:04:24.999756 27507 raft_consensus.cc:357] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:24.999847 27507 raft_consensus.cc:738] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:24.999987 27507 consensus_queue.cc:260] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.000288 27507 ts_tablet_manager.cc:1428] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:25.000437 27507 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:25.002954 27507 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 02:04:25.003175 27569 heartbeater.cc:344] Connected to a master server at 127.19.220.190:42333
I20250626 02:04:25.003242 27507 tablet_bootstrap.cc:492] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Bootstrap complete.
I20250626 02:04:25.003270 27569 heartbeater.cc:461] Registering TS with master...
I20250626 02:04:25.003404 27507 ts_tablet_manager.cc:1397] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250626 02:04:25.003512 27569 heartbeater.cc:507] Master 127.19.220.190:42333 requested a full tablet report, sending...
I20250626 02:04:25.003757 27507 raft_consensus.cc:357] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.003883 27507 raft_consensus.cc:738] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:25.003997 27507 consensus_queue.cc:260] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.004305 27507 ts_tablet_manager.cc:1428] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250626 02:04:25.004448 27507 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:25.004474 27040 ts_manager.cc:194] Re-registered known tserver with Master: 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353)
I20250626 02:04:25.006270 27040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38558
I20250626 02:04:25.007014 27507 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:25.007076 27569 heartbeater.cc:499] Master 127.19.220.190:42333 was elected leader, sending a full tablet report...
I20250626 02:04:25.007249 27507 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap complete.
I20250626 02:04:25.007411 27507 ts_tablet_manager.cc:1397] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:04:25.007795 27507 raft_consensus.cc:357] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.007881 27507 raft_consensus.cc:738] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:25.008008 27507 consensus_queue.cc:260] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.008307 27507 ts_tablet_manager.cc:1428] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250626 02:04:25.008448 27507 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:25.009809 27406 consensus_queue.cc:579] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Leader has been unable to successfully communicate with peer 7b20a67f201e4924a8f003667eeb604b for more than 1 seconds (1.567s)
I20250626 02:04:25.011838 27528 tablet_service.cc:1515] Processing DeleteTablet for tablet fc88aaa9948348f5b4cb8fb2c31a61c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40184
I20250626 02:04:25.011868 27532 tablet_service.cc:1515] Processing DeleteTablet for tablet d904a02cce9348ffae84798371b0f5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40184
I20250626 02:04:25.011879 27221 consensus_queue.cc:237] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:25.012166 27576 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:25.012297 27576 raft_consensus.cc:2241] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:25.012401 27576 raft_consensus.cc:2270] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:25.012918 27576 ts_tablet_manager.cc:1905] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:25.013294 27507 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:25.013518 27507 tablet_bootstrap.cc:492] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap complete.
I20250626 02:04:25.013660 27507 ts_tablet_manager.cc:1397] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250626 02:04:25.014070 27507 raft_consensus.cc:357] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.014133 27296 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:04:25.014178 27507 raft_consensus.cc:738] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:25.014297 27507 consensus_queue.cc:260] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.014600 27528 tablet_service.cc:1515] Processing DeleteTablet for tablet 4212c38370ab4dfe9720fd1f788bb791 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40184
I20250626 02:04:25.014640 27507 ts_tablet_manager.cc:1428] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:25.014782 27507 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
I20250626 02:04:25.014864 27406 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:04:25.015261 27576 ts_tablet_manager.cc:1918] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 02:04:25.015338 27576 log.cc:1199] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/fc88aaa9948348f5b4cb8fb2c31a61c7
I20250626 02:04:25.015559 27576 ts_tablet_manager.cc:1939] T fc88aaa9948348f5b4cb8fb2c31a61c7 P 7b20a67f201e4924a8f003667eeb604b: Deleting consensus metadata
I20250626 02:04:25.016287 27026 catalog_manager.cc:4928] TS 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): tablet fc88aaa9948348f5b4cb8fb2c31a61c7 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:25.016498 27406 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.017096 27576 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:25.017218 27576 raft_consensus.cc:2241] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:25.017285 27576 pending_rounds.cc:70] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Trying to abort 1 pending ops.
I20250626 02:04:25.017298 27541 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:04:25.017315 27576 pending_rounds.cc:77] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171700587947921408 op_type: NO_OP noop_request { }
I20250626 02:04:25.017383 27576 raft_consensus.cc:2887] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250626 02:04:25.017427 27576 raft_consensus.cc:2270] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:25.017706 27362 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:25.017969 27576 ts_tablet_manager.cc:1905] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:25.018774 27541 raft_consensus.cc:1215] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2] Dedup: 1.1->[1.2-1.2]
I20250626 02:04:25.019044 27296 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.019176 27541 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.020315 27576 ts_tablet_manager.cc:1918] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 02:04:25.021159 27040 catalog_manager.cc:5582] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 reported cstate change: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.020387 27576 log.cc:1199] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/d904a02cce9348ffae84798371b0f5ca
I20250626 02:04:25.021641 27507 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:25.021704 27576 ts_tablet_manager.cc:1939] T d904a02cce9348ffae84798371b0f5ca P 7b20a67f201e4924a8f003667eeb604b: Deleting consensus metadata
I20250626 02:04:25.021869 27507 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap complete.
I20250626 02:04:25.022024 27507 ts_tablet_manager.cc:1397] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250626 02:04:25.022343 27026 catalog_manager.cc:4928] TS 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): tablet d904a02cce9348ffae84798371b0f5ca (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:25.022452 27507 raft_consensus.cc:357] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.022552 27507 raft_consensus.cc:738] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:25.022665 27507 consensus_queue.cc:260] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.022754 27027 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4cf3aa09af90400190885a70f7c47362 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 02:04:25.022913 27507 ts_tablet_manager.cc:1428] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:25.023057 27507 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap starting.
W20250626 02:04:25.024725 27187 consensus_peers.cc:487] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): Couldn't send request to peer ecaca65e824c4aa48c4a06ec56b90eb6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cf3aa09af90400190885a70f7c47362. This is attempt 1: this message will repeat every 5th retry.
W20250626 02:04:25.025110 27026 catalog_manager.cc:4908] TS 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): delete failed for tablet 4212c38370ab4dfe9720fd1f788bb791 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4212c38370ab4dfe9720fd1f788bb791 already in progress: opening tablet
I20250626 02:04:25.025979 27507 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:25.026181 27507 tablet_bootstrap.cc:492] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Bootstrap complete.
I20250626 02:04:25.026340 27507 ts_tablet_manager.cc:1397] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:04:25.026698 27528 tablet_service.cc:1515] Processing DeleteTablet for tablet e7ba4f3151ce49fe8524ae6834dc64f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40184
I20250626 02:04:25.026762 27507 raft_consensus.cc:357] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.026850 27507 raft_consensus.cc:738] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b20a67f201e4924a8f003667eeb604b, State: Initialized, Role: FOLLOWER
I20250626 02:04:25.026907 27576 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:25.026966 27507 consensus_queue.cc:260] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.027021 27576 raft_consensus.cc:2241] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:25.027125 27576 raft_consensus.cc:2270] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:25.027231 27507 ts_tablet_manager.cc:1428] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250626 02:04:25.027514 27576 ts_tablet_manager.cc:1905] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:25.028452 27528 tablet_service.cc:1515] Processing DeleteTablet for tablet 4212c38370ab4dfe9720fd1f788bb791 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:04:24 UTC) from {username='slave'} at 127.0.0.1:40184
I20250626 02:04:25.029148 27576 ts_tablet_manager.cc:1918] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 02:04:25.029206 27576 log.cc:1199] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/e7ba4f3151ce49fe8524ae6834dc64f1
I20250626 02:04:25.029350 27576 ts_tablet_manager.cc:1939] T e7ba4f3151ce49fe8524ae6834dc64f1 P 7b20a67f201e4924a8f003667eeb604b: Deleting consensus metadata
I20250626 02:04:25.029810 27576 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:25.029855 27026 catalog_manager.cc:4928] TS 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): tablet e7ba4f3151ce49fe8524ae6834dc64f1 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:25.029935 27576 raft_consensus.cc:2241] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:25.030043 27576 raft_consensus.cc:2270] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:25.030431 27576 ts_tablet_manager.cc:1905] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:04:25.031811 27576 ts_tablet_manager.cc:1918] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 02:04:25.031869 27576 log.cc:1199] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/4212c38370ab4dfe9720fd1f788bb791
I20250626 02:04:25.032043 27576 ts_tablet_manager.cc:1939] T 4212c38370ab4dfe9720fd1f788bb791 P 7b20a67f201e4924a8f003667eeb604b: Deleting consensus metadata
I20250626 02:04:25.032689 27026 catalog_manager.cc:4928] TS 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): tablet 4212c38370ab4dfe9720fd1f788bb791 (table dugtrio [id=8584a8a5cba64b09ad964d0a61b36d59]) successfully deleted
I20250626 02:04:25.039249 27348 consensus_queue.cc:579] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Leader has been unable to successfully communicate with peer 7b20a67f201e4924a8f003667eeb604b for more than 1 seconds (1.337s)
I20250626 02:04:25.041695 27296 consensus_queue.cc:237] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:25.043198 27221 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:04:25.043573 27584 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:04:25.044678 27539 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:04:25.045238 27584 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:04:25.045887 27585 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.046203 27221 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
W20250626 02:04:25.046538 27262 consensus_peers.cc:487] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 -> Peer ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): Couldn't send request to peer ecaca65e824c4aa48c4a06ec56b90eb6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef6548b73bbc43229fc8f974b4165ac7. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:25.047595 27539 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.047765 27029 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ef6548b73bbc43229fc8f974b4165ac7 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 02:04:25.048517 27042 catalog_manager.cc:5582] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "822f01a34c91459993dd88632e7cfe41" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.051295 27296 consensus_queue.cc:237] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.052253 27221 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:25.052282 27539 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:25.052721 27348 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:25.052862 27585 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:25.054085 27584 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.054473 27221 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.054502 27539 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.055724 27029 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ef6548b73bbc43229fc8f974b4165ac7 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:25.056834 27041 catalog_manager.cc:5582] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 reported cstate change: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New cstate: current_term: 1 leader_uuid: "822f01a34c91459993dd88632e7cfe41" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:25.061841 27453 tablet_service.cc:1515] Processing DeleteTablet for tablet ef6548b73bbc43229fc8f974b4165ac7 with delete_type TABLET_DATA_TOMBSTONED (TS ecaca65e824c4aa48c4a06ec56b90eb6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48454
W20250626 02:04:25.062326 27029 catalog_manager.cc:4895] TS ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): delete failed for tablet ef6548b73bbc43229fc8f974b4165ac7 because the tablet was not found. No further retry: Not found: Tablet not found: ef6548b73bbc43229fc8f974b4165ac7
I20250626 02:04:25.063616 27221 consensus_queue.cc:237] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:25.064751 27296 raft_consensus.cc:1273] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:04:25.064848 27539 raft_consensus.cc:1273] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 02:04:25.065135 27187 consensus_peers.cc:487] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): Couldn't send request to peer ecaca65e824c4aa48c4a06ec56b90eb6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06faf0dbd13448d4a2bf7bb4edfce338. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:25.065161 27406 consensus_queue.cc:1035] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:25.065296 27362 consensus_queue.cc:1035] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 02:04:25.066642 27346 raft_consensus.cc:2953] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.067664 27539 raft_consensus.cc:2953] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.068377 27027 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 06faf0dbd13448d4a2bf7bb4edfce338 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250626 02:04:25.068529 27296 raft_consensus.cc:2953] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.068948 27042 catalog_manager.cc:5582] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:25.070868 27221 consensus_queue.cc:237] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:25.072041 27539 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:04:25.072065 27296 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:04:25.072431 27406 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 02:04:25.072556 27187 consensus_peers.cc:487] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): Couldn't send request to peer ecaca65e824c4aa48c4a06ec56b90eb6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72a83458cfda49a6a0fbff57a1371c22. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:25.072703 27406 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:04:25.073442 27221 consensus_queue.cc:237] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } }
I20250626 02:04:25.074491 27296 raft_consensus.cc:1273] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:25.074548 27541 raft_consensus.cc:1273] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:25.074873 27346 consensus_queue.cc:1035] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:25.075003 27362 consensus_queue.cc:1035] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:25.076009 27406 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.076460 27541 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.076838 27296 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.077162 27295 raft_consensus.cc:2953] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } }
I20250626 02:04:25.076838 27362 raft_consensus.cc:2953] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } }
I20250626 02:04:25.078634 27040 catalog_manager.cc:5582] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b reported cstate change: config changed from index -1 to 2, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:25.079111 27539 raft_consensus.cc:2953] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } }
I20250626 02:04:25.080443 27027 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 72a83458cfda49a6a0fbff57a1371c22 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250626 02:04:25.080595 27027 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 06faf0dbd13448d4a2bf7bb4edfce338 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:25.081037 27042 catalog_manager.cc:5582] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:25.083225 27453 tablet_service.cc:1515] Processing DeleteTablet for tablet 06faf0dbd13448d4a2bf7bb4edfce338 with delete_type TABLET_DATA_TOMBSTONED (TS ecaca65e824c4aa48c4a06ec56b90eb6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48454
W20250626 02:04:25.083561 27029 catalog_manager.cc:4895] TS ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): delete failed for tablet 06faf0dbd13448d4a2bf7bb4edfce338 because the tablet was not found. No further retry: Not found: Tablet not found: 06faf0dbd13448d4a2bf7bb4edfce338
I20250626 02:04:25.083554 27221 consensus_queue.cc:237] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.085253 27539 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:25.085402 27296 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:25.085564 27592 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:25.085743 27362 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:25.086894 27592 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.087224 27539 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.087291 27296 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.088395 27027 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 72a83458cfda49a6a0fbff57a1371c22 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:25.089284 27042 catalog_manager.cc:5582] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:25.091243 27453 tablet_service.cc:1515] Processing DeleteTablet for tablet 72a83458cfda49a6a0fbff57a1371c22 with delete_type TABLET_DATA_TOMBSTONED (TS ecaca65e824c4aa48c4a06ec56b90eb6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48454
W20250626 02:04:25.091626 27029 catalog_manager.cc:4895] TS ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): delete failed for tablet 72a83458cfda49a6a0fbff57a1371c22 because the tablet was not found. No further retry: Not found: Tablet not found: 72a83458cfda49a6a0fbff57a1371c22
W20250626 02:04:25.236068 27090 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table 7a13b057fc834f76b69fac55c008b9c6: could not find any suitable replica to move from server d26ebacebabe43d6822b5a63ff17c6e2 to server ecaca65e824c4aa48c4a06ec56b90eb6
I20250626 02:04:25.442947 27593 ts_tablet_manager.cc:927] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Initiating tablet copy from peer d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:25.443282 27593 tablet_copy_client.cc:323] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:35589
I20250626 02:04:25.446622 27231 tablet_copy_service.cc:140] P d26ebacebabe43d6822b5a63ff17c6e2: Received BeginTabletCopySession request for tablet 4cf3aa09af90400190885a70f7c47362 from peer ecaca65e824c4aa48c4a06ec56b90eb6 ({username='slave'} at 127.0.0.1:53212)
I20250626 02:04:25.446738 27231 tablet_copy_service.cc:161] P d26ebacebabe43d6822b5a63ff17c6e2: Beginning new tablet copy session on tablet 4cf3aa09af90400190885a70f7c47362 from peer ecaca65e824c4aa48c4a06ec56b90eb6 at {username='slave'} at 127.0.0.1:53212: session id = ecaca65e824c4aa48c4a06ec56b90eb6-4cf3aa09af90400190885a70f7c47362
I20250626 02:04:25.447507 27231 tablet_copy_source_session.cc:215] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:25.448351 27593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf3aa09af90400190885a70f7c47362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:25.450261 27593 tablet_copy_client.cc:806] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:25.450423 27593 tablet_copy_client.cc:670] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:25.451499 27593 tablet_copy_client.cc:538] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:25.452756 27593 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap starting.
I20250626 02:04:25.455760 27593 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:25.455968 27593 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap complete.
I20250626 02:04:25.456110 27593 ts_tablet_manager.cc:1397] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250626 02:04:25.456522 27593 raft_consensus.cc:357] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:25.456625 27593 raft_consensus.cc:738] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ecaca65e824c4aa48c4a06ec56b90eb6, State: Initialized, Role: LEARNER
I20250626 02:04:25.456753 27593 consensus_queue.cc:260] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:25.457087 27593 ts_tablet_manager.cc:1428] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 02:04:25.457170 27489 heartbeater.cc:499] Master 127.19.220.190:42333 was elected leader, sending a full tablet report...
I20250626 02:04:25.457607 27231 tablet_copy_service.cc:342] P d26ebacebabe43d6822b5a63ff17c6e2: Request end of tablet copy session ecaca65e824c4aa48c4a06ec56b90eb6-4cf3aa09af90400190885a70f7c47362 received from {username='slave'} at 127.0.0.1:53212
I20250626 02:04:25.457698 27231 tablet_copy_service.cc:434] P d26ebacebabe43d6822b5a63ff17c6e2: ending tablet copy session ecaca65e824c4aa48c4a06ec56b90eb6-4cf3aa09af90400190885a70f7c47362 on tablet 4cf3aa09af90400190885a70f7c47362 with peer ecaca65e824c4aa48c4a06ec56b90eb6
I20250626 02:04:25.619668 27221 consensus_queue.cc:237] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } }
I20250626 02:04:25.620797 27296 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:25.620815 27539 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 02:04:25.621217 27599 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:25.621366 27592 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 02:04:25.622790 27598 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.623205 27539 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.623270 27296 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } }
I20250626 02:04:25.624677 27027 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4cf3aa09af90400190885a70f7c47362 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:25.625133 27041 catalog_manager.cc:5582] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 2 to 3, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) evicted. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } }
I20250626 02:04:25.627626 27453 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cf3aa09af90400190885a70f7c47362 with delete_type TABLET_DATA_TOMBSTONED (TS ecaca65e824c4aa48c4a06ec56b90eb6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48454
I20250626 02:04:25.628278 27604 tablet_replica.cc:331] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: stopping tablet replica
I20250626 02:04:25.628396 27604 raft_consensus.cc:2241] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Raft consensus shutting down.
I20250626 02:04:25.628492 27604 raft_consensus.cc:2270] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Raft consensus is shut down!
I20250626 02:04:25.628970 27604 ts_tablet_manager.cc:1905] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:04:25.630728 27604 ts_tablet_manager.cc:1918] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250626 02:04:25.630791 27604 log.cc:1199] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-3-root/wals/4cf3aa09af90400190885a70f7c47362
I20250626 02:04:25.631223 27029 catalog_manager.cc:4928] TS ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): tablet 4cf3aa09af90400190885a70f7c47362 (table test-workload [id=7a13b057fc834f76b69fac55c008b9c6]) successfully deleted
I20250626 02:04:26.242197 27296 consensus_queue.cc:237] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.243618 27539 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:26.243649 27221 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250626 02:04:26.243746 27262 consensus_peers.cc:487] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 -> Peer ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): Couldn't send request to peer ecaca65e824c4aa48c4a06ec56b90eb6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef6548b73bbc43229fc8f974b4165ac7. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:26.244107 27609 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:26.244349 27585 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:26.245947 27609 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.246424 27221 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.246641 27539 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.248729 27042 catalog_manager.cc:5582] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "822f01a34c91459993dd88632e7cfe41" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.252813 27221 consensus_queue.cc:237] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.254101 27539 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:26.254221 27296 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250626 02:04:26.254292 27187 consensus_peers.cc:487] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): Couldn't send request to peer ecaca65e824c4aa48c4a06ec56b90eb6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:26.254451 27618 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:26.254622 27599 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:26.255931 27599 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.256457 27296 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.257899 27539 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.258289 27221 consensus_queue.cc:237] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.259944 27539 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 02:04:26.259862 27040 catalog_manager.cc:5582] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 reported cstate change: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.260278 27617 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:26.260733 27296 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250626 02:04:26.260950 27187 consensus_peers.cc:487] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 -> Peer ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132:33109): Couldn't send request to peer ecaca65e824c4aa48c4a06ec56b90eb6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72a83458cfda49a6a0fbff57a1371c22. This is attempt 1: this message will repeat every 5th retry.
I20250626 02:04:26.261143 27617 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 02:04:26.261757 27620 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.262220 27539 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.265070 27042 catalog_manager.cc:5582] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:26.266922 27295 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } } }
I20250626 02:04:26.660342 27624 ts_tablet_manager.cc:927] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: Initiating tablet copy from peer 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677)
I20250626 02:04:26.660748 27624 tablet_copy_client.cc:323] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.131:41677
I20250626 02:04:26.664186 27306 tablet_copy_service.cc:140] P 822f01a34c91459993dd88632e7cfe41: Received BeginTabletCopySession request for tablet ef6548b73bbc43229fc8f974b4165ac7 from peer ecaca65e824c4aa48c4a06ec56b90eb6 ({username='slave'} at 127.0.0.1:40890)
I20250626 02:04:26.664268 27306 tablet_copy_service.cc:161] P 822f01a34c91459993dd88632e7cfe41: Beginning new tablet copy session on tablet ef6548b73bbc43229fc8f974b4165ac7 from peer ecaca65e824c4aa48c4a06ec56b90eb6 at {username='slave'} at 127.0.0.1:40890: session id = ecaca65e824c4aa48c4a06ec56b90eb6-ef6548b73bbc43229fc8f974b4165ac7
I20250626 02:04:26.665109 27306 tablet_copy_source_session.cc:215] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:26.665918 27624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6548b73bbc43229fc8f974b4165ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:26.667831 27624 tablet_copy_client.cc:806] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:26.667994 27624 tablet_copy_client.cc:670] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:26.669021 27624 tablet_copy_client.cc:538] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:26.670333 27624 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap starting.
I20250626 02:04:26.674463 27624 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:26.674686 27624 tablet_bootstrap.cc:492] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap complete.
I20250626 02:04:26.674841 27624 ts_tablet_manager.cc:1397] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250626 02:04:26.675206 27624 raft_consensus.cc:357] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.675320 27624 raft_consensus.cc:738] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ecaca65e824c4aa48c4a06ec56b90eb6, State: Initialized, Role: LEARNER
I20250626 02:04:26.675427 27624 consensus_queue.cc:260] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.675810 27624 ts_tablet_manager.cc:1428] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250626 02:04:26.676229 27306 tablet_copy_service.cc:342] P 822f01a34c91459993dd88632e7cfe41: Request end of tablet copy session ecaca65e824c4aa48c4a06ec56b90eb6-ef6548b73bbc43229fc8f974b4165ac7 received from {username='slave'} at 127.0.0.1:40890
I20250626 02:04:26.676311 27306 tablet_copy_service.cc:434] P 822f01a34c91459993dd88632e7cfe41: ending tablet copy session ecaca65e824c4aa48c4a06ec56b90eb6-ef6548b73bbc43229fc8f974b4165ac7 on tablet ef6548b73bbc43229fc8f974b4165ac7 with peer ecaca65e824c4aa48c4a06ec56b90eb6
I20250626 02:04:26.723994 27624 ts_tablet_manager.cc:927] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: Initiating tablet copy from peer d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:26.724402 27624 tablet_copy_client.cc:323] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:35589
I20250626 02:04:26.724808 27231 tablet_copy_service.cc:140] P d26ebacebabe43d6822b5a63ff17c6e2: Received BeginTabletCopySession request for tablet 72a83458cfda49a6a0fbff57a1371c22 from peer ecaca65e824c4aa48c4a06ec56b90eb6 ({username='slave'} at 127.0.0.1:53212)
I20250626 02:04:26.724905 27231 tablet_copy_service.cc:161] P d26ebacebabe43d6822b5a63ff17c6e2: Beginning new tablet copy session on tablet 72a83458cfda49a6a0fbff57a1371c22 from peer ecaca65e824c4aa48c4a06ec56b90eb6 at {username='slave'} at 127.0.0.1:53212: session id = ecaca65e824c4aa48c4a06ec56b90eb6-72a83458cfda49a6a0fbff57a1371c22
I20250626 02:04:26.725703 27231 tablet_copy_source_session.cc:215] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:26.726503 27624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a83458cfda49a6a0fbff57a1371c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:26.728853 27624 tablet_copy_client.cc:806] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:26.729040 27624 tablet_copy_client.cc:670] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:26.729981 27624 tablet_copy_client.cc:538] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:26.731155 27624 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap starting.
I20250626 02:04:26.735144 27624 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:26.735354 27624 tablet_bootstrap.cc:492] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap complete.
I20250626 02:04:26.735517 27624 ts_tablet_manager.cc:1397] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20250626 02:04:26.735898 27624 raft_consensus.cc:357] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.736002 27624 raft_consensus.cc:738] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ecaca65e824c4aa48c4a06ec56b90eb6, State: Initialized, Role: LEARNER
I20250626 02:04:26.736148 27624 consensus_queue.cc:260] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.736461 27624 ts_tablet_manager.cc:1428] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 02:04:26.736882 27231 tablet_copy_service.cc:342] P d26ebacebabe43d6822b5a63ff17c6e2: Request end of tablet copy session ecaca65e824c4aa48c4a06ec56b90eb6-72a83458cfda49a6a0fbff57a1371c22 received from {username='slave'} at 127.0.0.1:53212
I20250626 02:04:26.736956 27231 tablet_copy_service.cc:434] P d26ebacebabe43d6822b5a63ff17c6e2: ending tablet copy session ecaca65e824c4aa48c4a06ec56b90eb6-72a83458cfda49a6a0fbff57a1371c22 on tablet 72a83458cfda49a6a0fbff57a1371c22 with peer ecaca65e824c4aa48c4a06ec56b90eb6
I20250626 02:04:26.828218 27624 ts_tablet_manager.cc:927] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Initiating tablet copy from peer d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589)
I20250626 02:04:26.828599 27624 tablet_copy_client.cc:323] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Beginning tablet copy session from remote peer at address 127.19.220.130:35589
I20250626 02:04:26.829010 27231 tablet_copy_service.cc:140] P d26ebacebabe43d6822b5a63ff17c6e2: Received BeginTabletCopySession request for tablet 4cf3aa09af90400190885a70f7c47362 from peer ecaca65e824c4aa48c4a06ec56b90eb6 ({username='slave'} at 127.0.0.1:53212)
I20250626 02:04:26.829108 27231 tablet_copy_service.cc:161] P d26ebacebabe43d6822b5a63ff17c6e2: Beginning new tablet copy session on tablet 4cf3aa09af90400190885a70f7c47362 from peer ecaca65e824c4aa48c4a06ec56b90eb6 at {username='slave'} at 127.0.0.1:53212: session id = ecaca65e824c4aa48c4a06ec56b90eb6-4cf3aa09af90400190885a70f7c47362
I20250626 02:04:26.829923 27231 tablet_copy_source_session.cc:215] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 02:04:26.830546 27624 ts_tablet_manager.cc:1905] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250626 02:04:26.832679 27624 ts_tablet_manager.cc:1918] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20250626 02:04:26.832866 27624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf3aa09af90400190885a70f7c47362. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:04:26.833912 27624 tablet_copy_client.cc:806] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Starting download of 0 data blocks...
I20250626 02:04:26.834046 27624 tablet_copy_client.cc:670] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Starting download of 1 WAL segments...
I20250626 02:04:26.835028 27624 tablet_copy_client.cc:538] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 02:04:26.836227 27624 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap starting.
I20250626 02:04:26.840093 27624 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:04:26.840368 27624 tablet_bootstrap.cc:492] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Bootstrap complete.
I20250626 02:04:26.840515 27624 ts_tablet_manager.cc:1397] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.001s
I20250626 02:04:26.840896 27624 raft_consensus.cc:357] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.841003 27624 raft_consensus.cc:738] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ecaca65e824c4aa48c4a06ec56b90eb6, State: Initialized, Role: LEARNER
I20250626 02:04:26.841117 27624 consensus_queue.cc:260] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: NON_VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: true } }
I20250626 02:04:26.841363 27624 ts_tablet_manager.cc:1428] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 02:04:26.841820 27231 tablet_copy_service.cc:342] P d26ebacebabe43d6822b5a63ff17c6e2: Request end of tablet copy session ecaca65e824c4aa48c4a06ec56b90eb6-4cf3aa09af90400190885a70f7c47362 received from {username='slave'} at 127.0.0.1:53212
I20250626 02:04:26.841884 27231 tablet_copy_service.cc:434] P d26ebacebabe43d6822b5a63ff17c6e2: ending tablet copy session ecaca65e824c4aa48c4a06ec56b90eb6-4cf3aa09af90400190885a70f7c47362 on tablet 4cf3aa09af90400190885a70f7c47362 with peer ecaca65e824c4aa48c4a06ec56b90eb6
I20250626 02:04:27.046706 27463 raft_consensus.cc:1215] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250626 02:04:27.315685 27463 raft_consensus.cc:1215] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250626 02:04:27.378134 27463 raft_consensus.cc:1215] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250626 02:04:27.552470 27634 raft_consensus.cc:1062] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41: attempting to promote NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 to VOTER
I20250626 02:04:27.552990 27634 consensus_queue.cc:237] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } }
I20250626 02:04:27.554278 27463 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.554394 27220 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.554713 27539 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.555112 27640 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.555393 27639 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.555601 27637 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.557914 27637 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.558142 27539 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.558087 27463 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.560453 27041 catalog_manager.cc:5582] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 reported cstate change: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "822f01a34c91459993dd88632e7cfe41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:27.561741 27221 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } attrs { replace: true } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.563723 27296 consensus_queue.cc:237] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } }
I20250626 02:04:27.564960 27463 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 02:04:27.565122 27539 raft_consensus.cc:1273] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer 822f01a34c91459993dd88632e7cfe41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 02:04:27.565328 27634 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 02:04:27.565622 27637 consensus_queue.cc:1035] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 02:04:27.567045 27634 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.567593 27463 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.568415 27539 raft_consensus.cc:2953] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.568773 27029 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ef6548b73bbc43229fc8f974b4165ac7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:27.569370 27040 catalog_manager.cc:5582] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 reported cstate change: config changed from index 5 to 6, VOTER d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130) evicted. New cstate: current_term: 1 leader_uuid: "822f01a34c91459993dd88632e7cfe41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:27.571920 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet ef6548b73bbc43229fc8f974b4165ac7 with delete_type TABLET_DATA_TOMBSTONED (TS d26ebacebabe43d6822b5a63ff17c6e2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53184
I20250626 02:04:27.572302 27649 tablet_replica.cc:331] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: stopping tablet replica
I20250626 02:04:27.572477 27649 raft_consensus.cc:2241] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:27.572692 27649 raft_consensus.cc:2270] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:27.573380 27649 ts_tablet_manager.cc:1905] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:04:27.575111 27649 ts_tablet_manager.cc:1918] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250626 02:04:27.575181 27649 log.cc:1199] T ef6548b73bbc43229fc8f974b4165ac7 P d26ebacebabe43d6822b5a63ff17c6e2: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-1-root/wals/ef6548b73bbc43229fc8f974b4165ac7
I20250626 02:04:27.575584 27027 catalog_manager.cc:4928] TS d26ebacebabe43d6822b5a63ff17c6e2 (127.19.220.130:35589): tablet ef6548b73bbc43229fc8f974b4165ac7 (table test-workload [id=7a13b057fc834f76b69fac55c008b9c6]) successfully deleted
I20250626 02:04:27.779291 27620 raft_consensus.cc:1062] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2: attempting to promote NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 to VOTER
I20250626 02:04:27.779767 27620 consensus_queue.cc:237] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } }
I20250626 02:04:27.781371 27463 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.781388 27296 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.781388 27539 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.781944 27651 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.782150 27635 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.782328 27636 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.784632 27635 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.785020 27539 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.786507 27463 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.786628 27296 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.786933 27041 catalog_manager.cc:5582] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:27.790423 27221 consensus_queue.cc:237] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } }
I20250626 02:04:27.791399 27463 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 02:04:27.791420 27296 raft_consensus.cc:1273] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 02:04:27.791791 27651 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 02:04:27.792083 27651 consensus_queue.cc:1035] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 02:04:27.793129 27620 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b20a67f201e4924a8f003667eeb604b (127.19.220.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.794679 27027 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 72a83458cfda49a6a0fbff57a1371c22 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:27.794790 27296 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b20a67f201e4924a8f003667eeb604b (127.19.220.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.795243 27463 raft_consensus.cc:2953] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7b20a67f201e4924a8f003667eeb604b (127.19.220.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.795260 27042 catalog_manager.cc:5582] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 5 to 6, VOTER 7b20a67f201e4924a8f003667eeb604b (127.19.220.129) evicted. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 02:04:27.797923 27528 tablet_service.cc:1515] Processing DeleteTablet for tablet 72a83458cfda49a6a0fbff57a1371c22 with delete_type TABLET_DATA_TOMBSTONED (TS 7b20a67f201e4924a8f003667eeb604b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40184
I20250626 02:04:27.798697 27655 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:27.798843 27655 raft_consensus.cc:2241] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:27.798998 27655 raft_consensus.cc:2270] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:27.799705 27655 ts_tablet_manager.cc:1905] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:04:27.802021 27655 ts_tablet_manager.cc:1918] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250626 02:04:27.802109 27655 log.cc:1199] T 72a83458cfda49a6a0fbff57a1371c22 P 7b20a67f201e4924a8f003667eeb604b: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-0-root/wals/72a83458cfda49a6a0fbff57a1371c22
I20250626 02:04:27.802614 27026 catalog_manager.cc:4928] TS 7b20a67f201e4924a8f003667eeb604b (127.19.220.129:37353): tablet 72a83458cfda49a6a0fbff57a1371c22 (table test-workload [id=7a13b057fc834f76b69fac55c008b9c6]) successfully deleted
I20250626 02:04:27.915853 27620 raft_consensus.cc:1062] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: attempting to promote NON_VOTER ecaca65e824c4aa48c4a06ec56b90eb6 to VOTER
I20250626 02:04:27.916316 27620 consensus_queue.cc:237] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } }
I20250626 02:04:27.917516 27539 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.917510 27463 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 LEARNER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.917510 27296 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 02:04:27.917912 27636 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.918128 27620 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.918267 27636 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 02:04:27.920099 27620 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.920487 27296 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.921566 27463 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.922462 27039 catalog_manager.cc:5582] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:27.923547 27539 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ecaca65e824c4aa48c4a06ec56b90eb6 (127.19.220.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "822f01a34c91459993dd88632e7cfe41" member_type: VOTER last_known_addr { host: "127.19.220.131" port: 41677 } attrs { replace: true } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.925719 27221 consensus_queue.cc:237] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } }
I20250626 02:04:27.926941 27463 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 02:04:27.927093 27539 raft_consensus.cc:1273] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Refusing update from remote peer d26ebacebabe43d6822b5a63ff17c6e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 02:04:27.927518 27651 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 02:04:27.927659 27620 consensus_queue.cc:1035] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 02:04:27.929265 27635 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 822f01a34c91459993dd88632e7cfe41 (127.19.220.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.929450 27463 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 822f01a34c91459993dd88632e7cfe41 (127.19.220.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.930459 27027 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4cf3aa09af90400190885a70f7c47362 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:04:27.930783 27539 raft_consensus.cc:2953] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 822f01a34c91459993dd88632e7cfe41 (127.19.220.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } } }
I20250626 02:04:27.931175 27042 catalog_manager.cc:5582] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 reported cstate change: config changed from index 5 to 6, VOTER 822f01a34c91459993dd88632e7cfe41 (127.19.220.131) evicted. New cstate: current_term: 1 leader_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7b20a67f201e4924a8f003667eeb604b" member_type: VOTER last_known_addr { host: "127.19.220.129" port: 37353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26ebacebabe43d6822b5a63ff17c6e2" member_type: VOTER last_known_addr { host: "127.19.220.130" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecaca65e824c4aa48c4a06ec56b90eb6" member_type: VOTER last_known_addr { host: "127.19.220.132" port: 33109 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 02:04:27.933375 27283 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cf3aa09af90400190885a70f7c47362 with delete_type TABLET_DATA_TOMBSTONED (TS 822f01a34c91459993dd88632e7cfe41 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40862
I20250626 02:04:27.933760 27660 tablet_replica.cc:331] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: stopping tablet replica
I20250626 02:04:27.933912 27660 raft_consensus.cc:2241] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:27.934072 27660 raft_consensus.cc:2270] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:27.934791 27660 ts_tablet_manager.cc:1905] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:04:27.936508 27660 ts_tablet_manager.cc:1918] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250626 02:04:27.936571 27660 log.cc:1199] T 4cf3aa09af90400190885a70f7c47362 P 822f01a34c91459993dd88632e7cfe41: Deleting WAL directory at /tmp/dist-test-task5Is7hp/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750903288526542-20338-0/minicluster-data/ts-2-root/wals/4cf3aa09af90400190885a70f7c47362
I20250626 02:04:27.937036 27029 catalog_manager.cc:4928] TS 822f01a34c91459993dd88632e7cfe41 (127.19.220.131:41677): tablet 4cf3aa09af90400190885a70f7c47362 (table test-workload [id=7a13b057fc834f76b69fac55c008b9c6]) successfully deleted
I20250626 02:04:30.024088 20338 tablet_server.cc:178] TabletServer@127.19.220.129:37353 shutting down...
I20250626 02:04:30.027575 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:30.027796 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:30.027918 20338 raft_consensus.cc:2241] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:30.028055 20338 raft_consensus.cc:2270] T 06faf0dbd13448d4a2bf7bb4edfce338 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.028540 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:30.028620 20338 raft_consensus.cc:2241] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:30.028733 20338 raft_consensus.cc:2270] T 4cf3aa09af90400190885a70f7c47362 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.029003 20338 tablet_replica.cc:331] stopping tablet replica
I20250626 02:04:30.029068 20338 raft_consensus.cc:2241] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:30.029162 20338 raft_consensus.cc:2270] T ef6548b73bbc43229fc8f974b4165ac7 P 7b20a67f201e4924a8f003667eeb604b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.041780 20338 tablet_server.cc:195] TabletServer@127.19.220.129:37353 shutdown complete.
I20250626 02:04:30.044380 20338 tablet_server.cc:178] TabletServer@127.19.220.130:0 shutting down...
I20250626 02:04:30.048113 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:30.048313 20338 tablet_replica.cc:331] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2: stopping tablet replica
I20250626 02:04:30.048429 20338 raft_consensus.cc:2241] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:30.048600 20338 raft_consensus.cc:2270] T 72a83458cfda49a6a0fbff57a1371c22 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.048975 20338 tablet_replica.cc:331] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2: stopping tablet replica
I20250626 02:04:30.049048 20338 raft_consensus.cc:2241] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:30.049176 20338 raft_consensus.cc:2270] T 4cf3aa09af90400190885a70f7c47362 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.049453 20338 tablet_replica.cc:331] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2: stopping tablet replica
I20250626 02:04:30.049538 20338 raft_consensus.cc:2241] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:30.049655 20338 raft_consensus.cc:2270] T 06faf0dbd13448d4a2bf7bb4edfce338 P d26ebacebabe43d6822b5a63ff17c6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.062382 20338 tablet_server.cc:195] TabletServer@127.19.220.130:0 shutdown complete.
I20250626 02:04:30.064771 20338 tablet_server.cc:178] TabletServer@127.19.220.131:0 shutting down...
I20250626 02:04:30.068449 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:30.068637 20338 tablet_replica.cc:331] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41: stopping tablet replica
I20250626 02:04:30.068758 20338 raft_consensus.cc:2241] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:30.068940 20338 raft_consensus.cc:2270] T ef6548b73bbc43229fc8f974b4165ac7 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.069307 20338 tablet_replica.cc:331] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41: stopping tablet replica
I20250626 02:04:30.069401 20338 raft_consensus.cc:2241] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:30.069528 20338 raft_consensus.cc:2270] T 72a83458cfda49a6a0fbff57a1371c22 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.069844 20338 tablet_replica.cc:331] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41: stopping tablet replica
I20250626 02:04:30.069932 20338 raft_consensus.cc:2241] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:30.070034 20338 raft_consensus.cc:2270] T 06faf0dbd13448d4a2bf7bb4edfce338 P 822f01a34c91459993dd88632e7cfe41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.082603 20338 tablet_server.cc:195] TabletServer@127.19.220.131:0 shutdown complete.
I20250626 02:04:30.084934 20338 tablet_server.cc:178] TabletServer@127.19.220.132:0 shutting down...
I20250626 02:04:30.088248 20338 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 02:04:30.088403 20338 tablet_replica.cc:331] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6: stopping tablet replica
I20250626 02:04:30.088537 20338 raft_consensus.cc:2241] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:30.088653 20338 raft_consensus.cc:2270] T 72a83458cfda49a6a0fbff57a1371c22 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.088986 20338 tablet_replica.cc:331] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6: stopping tablet replica
I20250626 02:04:30.089058 20338 raft_consensus.cc:2241] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:30.089141 20338 raft_consensus.cc:2270] T 4cf3aa09af90400190885a70f7c47362 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.089398 20338 tablet_replica.cc:331] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6: stopping tablet replica
I20250626 02:04:30.089460 20338 raft_consensus.cc:2241] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:04:30.089545 20338 raft_consensus.cc:2270] T ef6548b73bbc43229fc8f974b4165ac7 P ecaca65e824c4aa48c4a06ec56b90eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.101842 20338 tablet_server.cc:195] TabletServer@127.19.220.132:0 shutdown complete.
I20250626 02:04:30.103930 20338 master.cc:561] Master@127.19.220.190:42333 shutting down...
I20250626 02:04:30.107098 20338 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:04:30.107214 20338 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:04:30.107259 20338 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8ed32e6b9b6945d9a52be6717c9f7f8f: stopping tablet replica
I20250626 02:04:30.119658 20338 master.cc:583] Master@127.19.220.190:42333 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (7926 ms)
[----------] 14 tests from AutoRebalancerTest (181591 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (181591 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20250626 02:04:30.127822 20338 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20250626 02:04:30.127864 20338 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:536: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20250626 02:04:30.127882 20338 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:4671: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20250626 02:04:30.127902 20338 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5260: suppressed but not reported on 13 messages since previous log ~30 seconds ago
I20250626 02:04:30.127925 20338 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 2 messages since previous log ~167 seconds ago
I20250626 02:04:30.127948 20338 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 48 messages since previous log ~17 seconds ago
I20250626 02:04:30.128014 20338 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 24 messages since previous log ~8 seconds ago