[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250622 13:51:21.182091 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:39181,127.0.103.125:34163,127.0.103.124:44183
I20250622 13:51:21.184096 413 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:51:21.185300 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:21.196907 419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:21.196826 418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:21.196854 421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.197744 413 server_base.cc:1048] running on GCE node
I20250622 13:51:21.198581 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:21.198757 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:21.198843 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600281198822 us; error 0 us; skew 500 ppm
I20250622 13:51:21.199185 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:21.201860 413 webserver.cc:469] Webserver started at http://127.0.103.126:41341/ using document root <none> and password file <none>
I20250622 13:51:21.202584 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:21.202694 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:21.203081 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:21.206352 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "67d70696e3224ec4a5e79690f0e81661"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.212276 413 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250622 13:51:21.215502 426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.217468 413 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 13:51:21.217629 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "67d70696e3224ec4a5e79690f0e81661"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.217805 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:21.238161 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:21.239096 413 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:51:21.239418 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:21.261909 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39181
I20250622 13:51:21.261937 477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:39181 every 8 connection(s)
I20250622 13:51:21.264232 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:21.267184 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:21.268503 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.268604 478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250622 13:51:21.268844 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.275565 413 server_base.cc:1048] running on GCE node
I20250622 13:51:21.275885 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:21.275969 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:21.276052 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600281276013 us; error 0 us; skew 500 ppm
I20250622 13:51:21.276238 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:21.277741 413 webserver.cc:469] Webserver started at http://127.0.103.125:32777/ using document root <none> and password file <none>
I20250622 13:51:21.278004 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:21.278115 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:21.278272 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:21.278946 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root/instance:
uuid: "d008ca539514418da59fdcbc9b59ab73"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.281347 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:21.282176 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250622 13:51:21.283869 488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.284209 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:51:21.284314 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root
uuid: "d008ca539514418da59fdcbc9b59ab73"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.284507 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:21.297544 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:21.298349 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:21.301891 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
W20250622 13:51:21.303893 429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.125:34163: connect: Connection refused (error 111)
W20250622 13:51:21.306078 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.125:34163: Network error: Client connection negotiation failed: client connection to 127.0.103.125:34163: connect: Connection refused (error 111)
I20250622 13:51:21.315701 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.125:34163
I20250622 13:51:21.315721 542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.125:34163 every 8 connection(s)
I20250622 13:51:21.317126 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250622 13:51:21.317392 543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250622 13:51:21.320226 546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:21.320613 545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.321507 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:21.322728 413 server_base.cc:1048] running on GCE node
W20250622 13:51:21.322728 548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.323275 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:21.323366 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:21.323460 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600281323418 us; error 0 us; skew 500 ppm
I20250622 13:51:21.323662 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:21.325171 413 webserver.cc:469] Webserver started at http://127.0.103.124:34169/ using document root <none> and password file <none>
I20250622 13:51:21.325464 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:21.325575 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:21.325713 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:21.326432 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root/instance:
uuid: "4508d4f8e53c44cfa7311cc8d82d309d"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.329238 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:21.329577 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250622 13:51:21.331629 555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.332051 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:51:21.332166 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root
uuid: "4508d4f8e53c44cfa7311cc8d82d309d"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.332302 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:21.341672 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } has no permanent_uuid. Determining permanent_uuid...
W20250622 13:51:21.343827 543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.124:44183: Network error: Client connection negotiation failed: client connection to 127.0.103.124:44183: connect: Connection refused (error 111)
I20250622 13:51:21.355458 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } attempt: 1
I20250622 13:51:21.362462 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } has no permanent_uuid. Determining permanent_uuid...
W20250622 13:51:21.364097 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.124:44183: Network error: Client connection negotiation failed: client connection to 127.0.103.124:44183: connect: Connection refused (error 111)
I20250622 13:51:21.364859 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:21.365499 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:21.381153 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.124:44183
I20250622 13:51:21.381207 607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.124:44183 every 8 connection(s)
I20250622 13:51:21.382293 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:51:21.382987 608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:21.385500 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:21.392791 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:21.396046 543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } attempt: 1
I20250622 13:51:21.398658 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:21.407351 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } attempt: 1
I20250622 13:51:21.408991 543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73: Bootstrap starting.
I20250622 13:51:21.409060 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: Bootstrap starting.
I20250622 13:51:21.413770 608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:21.413780 543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:21.415588 608 log.cc:826] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: Log is configured to *not* fsync() on all Append() calls
I20250622 13:51:21.418469 543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73: No bootstrap required, opened a new log
I20250622 13:51:21.418553 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: No bootstrap required, opened a new log
I20250622 13:51:21.419083 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: Bootstrap starting.
I20250622 13:51:21.421142 478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:21.422873 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: No bootstrap required, opened a new log
I20250622 13:51:21.424738 478 raft_consensus.cc:357] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.424832 608 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.424804 543 raft_consensus.cc:357] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.425071 543 raft_consensus.cc:383] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:21.425031 478 raft_consensus.cc:383] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:21.425062 608 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:21.425295 543 raft_consensus.cc:738] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d008ca539514418da59fdcbc9b59ab73, State: Initialized, Role: FOLLOWER
I20250622 13:51:21.425387 478 raft_consensus.cc:738] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d70696e3224ec4a5e79690f0e81661, State: Initialized, Role: FOLLOWER
I20250622 13:51:21.425417 608 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4508d4f8e53c44cfa7311cc8d82d309d, State: Initialized, Role: FOLLOWER
I20250622 13:51:21.426149 478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.426155 543 consensus_queue.cc:260] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.426157 608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.427145 617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.427419 617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:21.427937 616 sys_catalog.cc:455] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.428202 616 sys_catalog.cc:458] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:21.428637 478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:51:21.428680 543 sys_catalog.cc:564] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:51:21.433261 618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.433573 608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:51:21.433573 618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:21.440501 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250622 13:51:21.443223 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250622 13:51:21.448269 636 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250622 13:51:21.448423 636 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250622 13:51:21.448246 643 catalog_manager.cc:1560] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250622 13:51:21.448573 643 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250622 13:51:21.454072 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:21.454699 652 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250622 13:51:21.454809 652 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250622 13:51:21.459148 653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:21.459228 654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:21.461122 656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.461295 413 server_base.cc:1048] running on GCE node
I20250622 13:51:21.461750 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:21.461853 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:21.461916 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600281461894 us; error 0 us; skew 500 ppm
I20250622 13:51:21.462106 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:21.464074 413 webserver.cc:469] Webserver started at http://127.0.103.65:35825/ using document root <none> and password file <none>
I20250622 13:51:21.464354 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:21.464447 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:21.464604 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:21.465302 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "9f245a1ee8014666ad316ba5514f0b39"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.468250 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250622 13:51:21.470144 661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.470537 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250622 13:51:21.470667 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "9f245a1ee8014666ad316ba5514f0b39"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.470790 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:21.476308 616 raft_consensus.cc:491] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:21.476631 616 raft_consensus.cc:513] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.478837 616 leader_election.cc:290] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67d70696e3224ec4a5e79690f0e81661 (127.0.103.126:39181), 4508d4f8e53c44cfa7311cc8d82d309d (127.0.103.124:44183)
I20250622 13:51:21.479696 583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d008ca539514418da59fdcbc9b59ab73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" is_pre_election: true
I20250622 13:51:21.480227 583 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d008ca539514418da59fdcbc9b59ab73 in term 0.
I20250622 13:51:21.481030 494 leader_election.cc:304] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4508d4f8e53c44cfa7311cc8d82d309d, d008ca539514418da59fdcbc9b59ab73; no voters:
I20250622 13:51:21.481637 616 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:21.481797 616 raft_consensus.cc:491] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:21.481909 616 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:21.482713 453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d008ca539514418da59fdcbc9b59ab73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d70696e3224ec4a5e79690f0e81661" is_pre_election: true
I20250622 13:51:21.483026 453 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d008ca539514418da59fdcbc9b59ab73 in term 0.
I20250622 13:51:21.485314 616 raft_consensus.cc:513] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.486135 616 leader_election.cc:290] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [CANDIDATE]: Term 1 election: Requested vote from peers 67d70696e3224ec4a5e79690f0e81661 (127.0.103.126:39181), 4508d4f8e53c44cfa7311cc8d82d309d (127.0.103.124:44183)
I20250622 13:51:21.486735 583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d008ca539514418da59fdcbc9b59ab73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4508d4f8e53c44cfa7311cc8d82d309d"
I20250622 13:51:21.486764 453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d008ca539514418da59fdcbc9b59ab73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d70696e3224ec4a5e79690f0e81661"
I20250622 13:51:21.486919 583 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:21.486937 453 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:21.490805 583 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d008ca539514418da59fdcbc9b59ab73 in term 1.
W20250622 13:51:21.491483 432 tablet.cc:2378] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:51:21.491484 494 leader_election.cc:304] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4508d4f8e53c44cfa7311cc8d82d309d, d008ca539514418da59fdcbc9b59ab73; no voters:
I20250622 13:51:21.492043 616 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:21.492383 453 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d008ca539514418da59fdcbc9b59ab73 in term 1.
I20250622 13:51:21.493681 616 raft_consensus.cc:695] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 1 LEADER]: Becoming Leader. State: Replica: d008ca539514418da59fdcbc9b59ab73, State: Running, Role: LEADER
I20250622 13:51:21.494410 616 consensus_queue.cc:237] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } }
I20250622 13:51:21.495321 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:21.496033 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:21.497653 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:21.499461 666 sys_catalog.cc:455] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d008ca539514418da59fdcbc9b59ab73. Latest consensus state: current_term: 1 leader_uuid: "d008ca539514418da59fdcbc9b59ab73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.499768 666 sys_catalog.cc:458] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:21.501528 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:21.501649 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.501771 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:21.501856 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.510732 670 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:51:21.514318 670 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:51:21.527304 453 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 1 FOLLOWER]: Refusing update from remote peer d008ca539514418da59fdcbc9b59ab73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:21.527297 583 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 1 FOLLOWER]: Refusing update from remote peer d008ca539514418da59fdcbc9b59ab73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:21.528494 616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:21.528857 666 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:21.535223 618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d008ca539514418da59fdcbc9b59ab73. Latest consensus state: current_term: 1 leader_uuid: "d008ca539514418da59fdcbc9b59ab73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.535566 618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:21.540748 701 mvcc.cc:204] Tried to move back new op lower bound from 7170458753121075200 to 7170458753007087616. Current Snapshot: MvccSnapshot[applied={T|T < 7170458753121075200}]
I20250622 13:51:21.541201 618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d008ca539514418da59fdcbc9b59ab73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.541433 618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:21.544476 616 sys_catalog.cc:455] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d008ca539514418da59fdcbc9b59ab73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.544776 616 sys_catalog.cc:458] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:21.545390 670 catalog_manager.cc:1349] Generated new cluster ID: da2bcccd03c446eba9b5dbe5c7d8db89
I20250622 13:51:21.545523 670 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:51:21.553455 617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d008ca539514418da59fdcbc9b59ab73. Latest consensus state: current_term: 1 leader_uuid: "d008ca539514418da59fdcbc9b59ab73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.553720 617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:21.555063 666 sys_catalog.cc:455] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d008ca539514418da59fdcbc9b59ab73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.555291 666 sys_catalog.cc:458] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:21.573241 617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d008ca539514418da59fdcbc9b59ab73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d70696e3224ec4a5e79690f0e81661" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39181 } } peers { permanent_uuid: "d008ca539514418da59fdcbc9b59ab73" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 34163 } } peers { permanent_uuid: "4508d4f8e53c44cfa7311cc8d82d309d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 44183 } } }
I20250622 13:51:21.573511 617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:21.587198 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39341
I20250622 13:51:21.587230 734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39341 every 8 connection(s)
I20250622 13:51:21.616386 670 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:51:21.618866 670 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:51:21.641079 736 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39181
I20250622 13:51:21.641431 736 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.642720 736 heartbeater.cc:507] Master 127.0.103.126:39181 requested a full tablet report, sending...
I20250622 13:51:21.643404 735 heartbeater.cc:344] Connected to a master server at 127.0.103.124:44183
I20250622 13:51:21.643550 735 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.644121 735 heartbeater.cc:507] Master 127.0.103.124:44183 requested a full tablet report, sending...
I20250622 13:51:21.647177 573 ts_manager.cc:194] Registered new tserver with Master: 9f245a1ee8014666ad316ba5514f0b39 (127.0.103.65:39341)
I20250622 13:51:21.649709 443 ts_manager.cc:194] Registered new tserver with Master: 9f245a1ee8014666ad316ba5514f0b39 (127.0.103.65:39341)
I20250622 13:51:21.652895 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250622 13:51:21.659584 738 heartbeater.cc:344] Connected to a master server at 127.0.103.125:34163
I20250622 13:51:21.659803 738 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.660632 738 heartbeater.cc:507] Master 127.0.103.125:34163 requested a full tablet report, sending...
I20250622 13:51:21.662019 508 ts_manager.cc:194] Registered new tserver with Master: 9f245a1ee8014666ad316ba5514f0b39 (127.0.103.65:39341)
W20250622 13:51:21.663681 742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:21.665328 743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.670271 670 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73: Generated new TSK 0
I20250622 13:51:21.671221 670 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:51:21.672845 413 server_base.cc:1048] running on GCE node
W20250622 13:51:21.673610 745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.674037 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:21.674127 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:21.674211 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600281674172 us; error 0 us; skew 500 ppm
I20250622 13:51:21.674381 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:21.676054 413 webserver.cc:469] Webserver started at http://127.0.103.66:34093/ using document root <none> and password file <none>
I20250622 13:51:21.676276 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:21.676380 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:21.676532 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:21.677197 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "c306c9a6bcff4ceebc5f68de5bb835e7"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.679996 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250622 13:51:21.681864 750 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.682289 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:51:21.682399 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "c306c9a6bcff4ceebc5f68de5bb835e7"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.682552 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:21.692958 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:21.693687 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:21.694370 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:21.695431 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:21.695502 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.695616 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:21.695672 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.719801 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:43421
I20250622 13:51:21.719853 812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:43421 every 8 connection(s)
I20250622 13:51:21.722827 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:21.746420 819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:21.752506 821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.759624 813 heartbeater.cc:344] Connected to a master server at 127.0.103.124:44183
I20250622 13:51:21.760010 813 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.763435 815 heartbeater.cc:344] Connected to a master server at 127.0.103.125:34163
I20250622 13:51:21.763607 815 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.763865 413 server_base.cc:1048] running on GCE node
I20250622 13:51:21.765269 813 heartbeater.cc:507] Master 127.0.103.124:44183 requested a full tablet report, sending...
I20250622 13:51:21.765555 815 heartbeater.cc:507] Master 127.0.103.125:34163 requested a full tablet report, sending...
I20250622 13:51:21.766528 508 ts_manager.cc:194] Registered new tserver with Master: c306c9a6bcff4ceebc5f68de5bb835e7 (127.0.103.66:43421)
I20250622 13:51:21.766863 573 ts_manager.cc:194] Registered new tserver with Master: c306c9a6bcff4ceebc5f68de5bb835e7 (127.0.103.66:43421)
I20250622 13:51:21.767024 814 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39181
I20250622 13:51:21.767133 814 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.767711 814 heartbeater.cc:507] Master 127.0.103.126:39181 requested a full tablet report, sending...
I20250622 13:51:21.768707 443 ts_manager.cc:194] Registered new tserver with Master: c306c9a6bcff4ceebc5f68de5bb835e7 (127.0.103.66:43421)
I20250622 13:51:21.768725 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33048
W20250622 13:51:21.770104 823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:21.770614 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:21.770730 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:21.770794 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600281770772 us; error 0 us; skew 500 ppm
I20250622 13:51:21.770987 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:21.772580 413 webserver.cc:469] Webserver started at http://127.0.103.67:38103/ using document root <none> and password file <none>
I20250622 13:51:21.772850 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:21.772931 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:21.773101 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:21.773805 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "0af074ae30674189993a69766873dca3"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.777038 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250622 13:51:21.778975 828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.779385 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250622 13:51:21.779497 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "0af074ae30674189993a69766873dca3"
format_stamp: "Formatted at 2025-06-22 13:51:21 on dist-test-slave-tmv0"
I20250622 13:51:21.779614 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:21.801007 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:21.801736 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:21.802414 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:21.803552 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:21.803639 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.803745 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:21.803833 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:21.827028 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44889
I20250622 13:51:21.827049 890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44889 every 8 connection(s)
I20250622 13:51:21.838963 891 heartbeater.cc:344] Connected to a master server at 127.0.103.124:44183
I20250622 13:51:21.839164 891 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.839859 891 heartbeater.cc:507] Master 127.0.103.124:44183 requested a full tablet report, sending...
I20250622 13:51:21.841017 573 ts_manager.cc:194] Registered new tserver with Master: 0af074ae30674189993a69766873dca3 (127.0.103.67:44889)
I20250622 13:51:21.845115 893 heartbeater.cc:344] Connected to a master server at 127.0.103.125:34163
I20250622 13:51:21.845278 893 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.845944 893 heartbeater.cc:507] Master 127.0.103.125:34163 requested a full tablet report, sending...
I20250622 13:51:21.846041 892 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39181
I20250622 13:51:21.846158 892 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:21.846640 892 heartbeater.cc:507] Master 127.0.103.126:39181 requested a full tablet report, sending...
I20250622 13:51:21.846956 508 ts_manager.cc:194] Registered new tserver with Master: 0af074ae30674189993a69766873dca3 (127.0.103.67:44889)
I20250622 13:51:21.847716 443 ts_manager.cc:194] Registered new tserver with Master: 0af074ae30674189993a69766873dca3 (127.0.103.67:44889)
I20250622 13:51:21.848024 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012022192s
I20250622 13:51:21.848639 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33062
I20250622 13:51:22.450284 636 catalog_manager.cc:1261] Loaded cluster ID: da2bcccd03c446eba9b5dbe5c7d8db89
I20250622 13:51:22.450390 636 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: loading cluster ID for follower catalog manager: success
I20250622 13:51:22.452390 636 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: acquiring CA information for follower catalog manager: success
I20250622 13:51:22.453569 636 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250622 13:51:22.455772 652 catalog_manager.cc:1261] Loaded cluster ID: da2bcccd03c446eba9b5dbe5c7d8db89
I20250622 13:51:22.455837 652 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: loading cluster ID for follower catalog manager: success
I20250622 13:51:22.458024 652 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: acquiring CA information for follower catalog manager: success
I20250622 13:51:22.458909 652 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250622 13:51:22.664744 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33040
I20250622 13:51:22.770857 815 heartbeater.cc:499] Master 127.0.103.125:34163 was elected leader, sending a full tablet report...
I20250622 13:51:22.850032 893 heartbeater.cc:499] Master 127.0.103.125:34163 was elected leader, sending a full tablet report...
I20250622 13:51:23.443126 642 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:23.666075 738 heartbeater.cc:499] Master 127.0.103.125:34163 was elected leader, sending a full tablet report...
I20250622 13:51:23.919683 413 test_util.cc:276] Using random seed: 1678841128
I20250622 13:51:23.971159 508 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33078:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 13:51:24.028213 856 tablet_service.cc:1468] Processing CreateTablet for tablet 7e3eb55f3c914655b70a246d78f62476 (DEFAULT_TABLE table=test-workload [id=10d123a211b34932826cb999bf6a7c23]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:51:24.029549 693 tablet_service.cc:1468] Processing CreateTablet for tablet 068545eacf4148d296f7995bffc1bce5 (DEFAULT_TABLE table=test-workload [id=10d123a211b34932826cb999bf6a7c23]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:51:24.030393 693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068545eacf4148d296f7995bffc1bce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:24.035213 856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e3eb55f3c914655b70a246d78f62476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:24.035059 692 tablet_service.cc:1468] Processing CreateTablet for tablet f817d91c26474d8295580a0e6a3c5ff3 (DEFAULT_TABLE table=test-workload [id=10d123a211b34932826cb999bf6a7c23]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:51:24.035132 778 tablet_service.cc:1468] Processing CreateTablet for tablet 727af60386574cf3a2d4b39c0ca68297 (DEFAULT_TABLE table=test-workload [id=10d123a211b34932826cb999bf6a7c23]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:51:24.035665 926 tablet_bootstrap.cc:492] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39: Bootstrap starting.
I20250622 13:51:24.035672 692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f817d91c26474d8295580a0e6a3c5ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:24.035744 778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 727af60386574cf3a2d4b39c0ca68297. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:24.039613 926 tablet_bootstrap.cc:654] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:24.046027 928 tablet_bootstrap.cc:492] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3: Bootstrap starting.
I20250622 13:51:24.048691 928 tablet_bootstrap.cc:654] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:24.051137 929 tablet_bootstrap.cc:492] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7: Bootstrap starting.
I20250622 13:51:24.051378 926 tablet_bootstrap.cc:492] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39: No bootstrap required, opened a new log
I20250622 13:51:24.051549 926 ts_tablet_manager.cc:1397] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20250622 13:51:24.052419 926 raft_consensus.cc:357] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } }
I20250622 13:51:24.052618 926 raft_consensus.cc:383] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:24.052716 926 raft_consensus.cc:738] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f245a1ee8014666ad316ba5514f0b39, State: Initialized, Role: FOLLOWER
I20250622 13:51:24.052987 926 consensus_queue.cc:260] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } }
I20250622 13:51:24.053217 926 raft_consensus.cc:397] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:24.053267 929 tablet_bootstrap.cc:654] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:24.053315 926 raft_consensus.cc:491] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:24.053685 926 raft_consensus.cc:3058] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:24.055487 929 tablet_bootstrap.cc:492] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7: No bootstrap required, opened a new log
I20250622 13:51:24.055639 928 tablet_bootstrap.cc:492] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3: No bootstrap required, opened a new log
I20250622 13:51:24.055640 929 ts_tablet_manager.cc:1397] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250622 13:51:24.055838 928 ts_tablet_manager.cc:1397] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250622 13:51:24.056375 926 raft_consensus.cc:513] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } }
I20250622 13:51:24.056600 928 raft_consensus.cc:357] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af074ae30674189993a69766873dca3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44889 } }
I20250622 13:51:24.057130 928 raft_consensus.cc:383] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:24.057127 929 raft_consensus.cc:357] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c306c9a6bcff4ceebc5f68de5bb835e7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43421 } }
I20250622 13:51:24.057129 926 leader_election.cc:304] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f245a1ee8014666ad316ba5514f0b39; no voters:
I20250622 13:51:24.057308 929 raft_consensus.cc:383] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:24.057305 928 raft_consensus.cc:738] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af074ae30674189993a69766873dca3, State: Initialized, Role: FOLLOWER
I20250622 13:51:24.057466 929 raft_consensus.cc:738] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c306c9a6bcff4ceebc5f68de5bb835e7, State: Initialized, Role: FOLLOWER
I20250622 13:51:24.057680 928 consensus_queue.cc:260] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af074ae30674189993a69766873dca3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44889 } }
I20250622 13:51:24.057734 929 consensus_queue.cc:260] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c306c9a6bcff4ceebc5f68de5bb835e7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43421 } }
I20250622 13:51:24.057883 928 raft_consensus.cc:397] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:24.057909 929 raft_consensus.cc:397] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:24.057992 928 raft_consensus.cc:491] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:24.057996 929 raft_consensus.cc:491] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:24.058151 928 raft_consensus.cc:3058] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:24.058176 929 raft_consensus.cc:3058] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:24.060940 929 raft_consensus.cc:513] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c306c9a6bcff4ceebc5f68de5bb835e7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43421 } }
I20250622 13:51:24.061174 929 leader_election.cc:304] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c306c9a6bcff4ceebc5f68de5bb835e7; no voters:
I20250622 13:51:24.061173 928 raft_consensus.cc:513] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af074ae30674189993a69766873dca3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44889 } }
I20250622 13:51:24.061426 928 leader_election.cc:304] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af074ae30674189993a69766873dca3; no voters:
I20250622 13:51:24.061625 926 leader_election.cc:290] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:24.061782 932 raft_consensus.cc:2802] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:24.067953 933 raft_consensus.cc:2802] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:24.068166 929 leader_election.cc:290] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:24.069175 926 ts_tablet_manager.cc:1428] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39: Time spent starting tablet: real 0.018s user 0.000s sys 0.016s
I20250622 13:51:24.068703 928 leader_election.cc:290] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:24.069770 932 raft_consensus.cc:695] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 LEADER]: Becoming Leader. State: Replica: 9f245a1ee8014666ad316ba5514f0b39, State: Running, Role: LEADER
I20250622 13:51:24.068576 934 raft_consensus.cc:2802] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:24.070168 932 consensus_queue.cc:237] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } }
I20250622 13:51:24.070624 933 raft_consensus.cc:695] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 1 LEADER]: Becoming Leader. State: Replica: c306c9a6bcff4ceebc5f68de5bb835e7, State: Running, Role: LEADER
I20250622 13:51:24.071574 926 tablet_bootstrap.cc:492] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39: Bootstrap starting.
I20250622 13:51:24.072043 928 ts_tablet_manager.cc:1428] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20250622 13:51:24.070964 933 consensus_queue.cc:237] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c306c9a6bcff4ceebc5f68de5bb835e7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43421 } }
I20250622 13:51:24.073977 929 ts_tablet_manager.cc:1428] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7: Time spent starting tablet: real 0.018s user 0.008s sys 0.005s
I20250622 13:51:24.074414 926 tablet_bootstrap.cc:654] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:24.074376 934 raft_consensus.cc:695] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 1 LEADER]: Becoming Leader. State: Replica: 0af074ae30674189993a69766873dca3, State: Running, Role: LEADER
I20250622 13:51:24.074911 934 consensus_queue.cc:237] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af074ae30674189993a69766873dca3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44889 } }
I20250622 13:51:24.082433 508 catalog_manager.cc:5582] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f245a1ee8014666ad316ba5514f0b39 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9f245a1ee8014666ad316ba5514f0b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:24.091504 507 catalog_manager.cc:5582] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0af074ae30674189993a69766873dca3 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "0af074ae30674189993a69766873dca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af074ae30674189993a69766873dca3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44889 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:24.103554 506 catalog_manager.cc:5582] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c306c9a6bcff4ceebc5f68de5bb835e7 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "c306c9a6bcff4ceebc5f68de5bb835e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c306c9a6bcff4ceebc5f68de5bb835e7" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43421 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:24.106398 926 tablet_bootstrap.cc:492] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39: No bootstrap required, opened a new log
I20250622 13:51:24.106585 926 ts_tablet_manager.cc:1397] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.001s
I20250622 13:51:24.107445 926 raft_consensus.cc:357] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } }
I20250622 13:51:24.108043 926 raft_consensus.cc:383] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:24.108158 926 raft_consensus.cc:738] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f245a1ee8014666ad316ba5514f0b39, State: Initialized, Role: FOLLOWER
I20250622 13:51:24.108433 926 consensus_queue.cc:260] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } }
I20250622 13:51:24.108668 926 raft_consensus.cc:397] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:24.108800 926 raft_consensus.cc:491] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:24.108922 926 raft_consensus.cc:3058] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:24.111639 926 raft_consensus.cc:513] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } }
I20250622 13:51:24.112308 926 leader_election.cc:304] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f245a1ee8014666ad316ba5514f0b39; no voters:
I20250622 13:51:24.112519 926 leader_election.cc:290] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:24.112548 932 raft_consensus.cc:2802] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:24.112748 932 raft_consensus.cc:695] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 LEADER]: Becoming Leader. State: Replica: 9f245a1ee8014666ad316ba5514f0b39, State: Running, Role: LEADER
I20250622 13:51:24.112957 932 consensus_queue.cc:237] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } }
I20250622 13:51:24.114895 926 ts_tablet_manager.cc:1428] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20250622 13:51:24.115295 508 catalog_manager.cc:5582] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f245a1ee8014666ad316ba5514f0b39 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9f245a1ee8014666ad316ba5514f0b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f245a1ee8014666ad316ba5514f0b39" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39341 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:24.146703 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:51:24.157109 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:24.157545 413 tablet_replica.cc:331] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39: stopping tablet replica
I20250622 13:51:24.157943 413 raft_consensus.cc:2241] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:24.158228 413 raft_consensus.cc:2270] T f817d91c26474d8295580a0e6a3c5ff3 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:24.159804 413 tablet_replica.cc:331] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39: stopping tablet replica
I20250622 13:51:24.159967 413 raft_consensus.cc:2241] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:24.160123 413 raft_consensus.cc:2270] T 068545eacf4148d296f7995bffc1bce5 P 9f245a1ee8014666ad316ba5514f0b39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:24.165287 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:51:24.170220 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:51:24.178988 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:24.179342 413 tablet_replica.cc:331] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7: stopping tablet replica
I20250622 13:51:24.179503 413 raft_consensus.cc:2241] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:24.179656 413 raft_consensus.cc:2270] T 727af60386574cf3a2d4b39c0ca68297 P c306c9a6bcff4ceebc5f68de5bb835e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:24.193962 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:51:24.196622 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:51:24.205191 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:24.205598 413 tablet_replica.cc:331] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3: stopping tablet replica
I20250622 13:51:24.205822 413 raft_consensus.cc:2241] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:24.206005 413 raft_consensus.cc:2270] T 7e3eb55f3c914655b70a246d78f62476 P 0af074ae30674189993a69766873dca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:24.220566 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:51:24.223687 413 master.cc:561] Master@127.0.103.126:39181 shutting down...
I20250622 13:51:24.232827 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:24.233078 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:24.233214 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 67d70696e3224ec4a5e79690f0e81661: stopping tablet replica
I20250622 13:51:24.256007 413 master.cc:583] Master@127.0.103.126:39181 shutdown complete.
I20250622 13:51:24.263212 413 master.cc:561] Master@127.0.103.125:34163 shutting down...
I20250622 13:51:24.270871 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:24.271337 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:24.271453 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P d008ca539514418da59fdcbc9b59ab73: stopping tablet replica
I20250622 13:51:24.286825 413 master.cc:583] Master@127.0.103.125:34163 shutdown complete.
I20250622 13:51:24.290721 413 master.cc:561] Master@127.0.103.124:44183 shutting down...
I20250622 13:51:24.297757 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:24.297948 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:24.298059 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 4508d4f8e53c44cfa7311cc8d82d309d: stopping tablet replica
I20250622 13:51:24.312105 413 master.cc:583] Master@127.0.103.124:44183 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3145 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20250622 13:51:24.325168 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:35365
I20250622 13:51:24.325645 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:24.327778 949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:24.327960 950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:24.328460 952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:24.329020 413 server_base.cc:1048] running on GCE node
I20250622 13:51:24.329275 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:24.329370 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:24.329432 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600284329413 us; error 0 us; skew 500 ppm
I20250622 13:51:24.329619 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:24.334224 413 webserver.cc:469] Webserver started at http://127.0.103.126:42465/ using document root <none> and password file <none>
I20250622 13:51:24.334466 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:24.334551 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:24.334702 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:24.335364 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "d6a8e62d743a4526861a52bdd8454d6d"
format_stamp: "Formatted at 2025-06-22 13:51:24 on dist-test-slave-tmv0"
I20250622 13:51:24.337785 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20250622 13:51:24.339387 957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.339790 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:51:24.339884 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "d6a8e62d743a4526861a52bdd8454d6d"
format_stamp: "Formatted at 2025-06-22 13:51:24 on dist-test-slave-tmv0"
I20250622 13:51:24.339977 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:24.374270 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:24.374848 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:24.388442 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:35365
I20250622 13:51:24.388474 1008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:35365 every 8 connection(s)
I20250622 13:51:24.389853 1009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:24.393729 1009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d: Bootstrap starting.
I20250622 13:51:24.395043 1009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:24.396548 1009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d: No bootstrap required, opened a new log
I20250622 13:51:24.397300 1009 raft_consensus.cc:357] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a8e62d743a4526861a52bdd8454d6d" member_type: VOTER }
I20250622 13:51:24.397507 1009 raft_consensus.cc:383] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:24.397603 1009 raft_consensus.cc:738] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a8e62d743a4526861a52bdd8454d6d, State: Initialized, Role: FOLLOWER
I20250622 13:51:24.397876 1009 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a8e62d743a4526861a52bdd8454d6d" member_type: VOTER }
I20250622 13:51:24.398063 1009 raft_consensus.cc:397] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:24.398156 1009 raft_consensus.cc:491] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:24.398274 1009 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:24.399943 1009 raft_consensus.cc:513] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a8e62d743a4526861a52bdd8454d6d" member_type: VOTER }
I20250622 13:51:24.400170 1009 leader_election.cc:304] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a8e62d743a4526861a52bdd8454d6d; no voters:
I20250622 13:51:24.400547 1009 leader_election.cc:290] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:24.400666 1012 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:24.401188 1012 raft_consensus.cc:695] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 1 LEADER]: Becoming Leader. State: Replica: d6a8e62d743a4526861a52bdd8454d6d, State: Running, Role: LEADER
I20250622 13:51:24.401484 1009 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:51:24.401504 1012 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a8e62d743a4526861a52bdd8454d6d" member_type: VOTER }
I20250622 13:51:24.402793 1014 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a8e62d743a4526861a52bdd8454d6d. Latest consensus state: current_term: 1 leader_uuid: "d6a8e62d743a4526861a52bdd8454d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a8e62d743a4526861a52bdd8454d6d" member_type: VOTER } }
I20250622 13:51:24.402771 1013 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a8e62d743a4526861a52bdd8454d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a8e62d743a4526861a52bdd8454d6d" member_type: VOTER } }
I20250622 13:51:24.403280 1013 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:24.403534 1014 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:24.404384 1020 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:51:24.405910 1020 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:51:24.408066 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:51:24.408918 1020 catalog_manager.cc:1349] Generated new cluster ID: d275f031d8f84618b217743aa7b5292b
I20250622 13:51:24.409003 1020 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:51:24.416297 1020 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:51:24.416963 1020 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:51:24.424475 1020 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d: Generated new TSK 0
I20250622 13:51:24.424670 1020 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:51:24.441175 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:24.444327 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:24.444707 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:24.445247 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:24.445472 413 server_base.cc:1048] running on GCE node
I20250622 13:51:24.445699 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:24.445753 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:24.445803 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600284445779 us; error 0 us; skew 500 ppm
I20250622 13:51:24.445924 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:24.447113 413 webserver.cc:469] Webserver started at http://127.0.103.65:44741/ using document root <none> and password file <none>
I20250622 13:51:24.447301 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:24.447379 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:24.447497 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:24.448130 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "e613b3722f774902b7f3f2569a18c925"
format_stamp: "Formatted at 2025-06-22 13:51:24 on dist-test-slave-tmv0"
I20250622 13:51:24.450567 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250622 13:51:24.452083 1038 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.452436 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:51:24.452545 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "e613b3722f774902b7f3f2569a18c925"
format_stamp: "Formatted at 2025-06-22 13:51:24 on dist-test-slave-tmv0"
I20250622 13:51:24.452684 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:24.471529 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:24.472121 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:24.472651 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:24.473513 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:24.473595 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.473695 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:24.473744 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.488428 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:46271
I20250622 13:51:24.488463 1100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:46271 every 8 connection(s)
I20250622 13:51:24.490118 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:24.495693 1105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:24.496719 1106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:24.497545 413 server_base.cc:1048] running on GCE node
W20250622 13:51:24.498030 1108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:24.498409 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:24.498512 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:24.498584 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600284498555 us; error 0 us; skew 500 ppm
I20250622 13:51:24.498754 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:24.500627 413 webserver.cc:469] Webserver started at http://127.0.103.66:35309/ using document root <none> and password file <none>
I20250622 13:51:24.500880 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:24.500969 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:24.501134 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:24.502246 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "72d517046abc4f898ddb321158a611d3"
format_stamp: "Formatted at 2025-06-22 13:51:24 on dist-test-slave-tmv0"
I20250622 13:51:24.502562 1101 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35365
I20250622 13:51:24.502681 1101 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:24.503091 1101 heartbeater.cc:507] Master 127.0.103.126:35365 requested a full tablet report, sending...
I20250622 13:51:24.503962 974 ts_manager.cc:194] Registered new tserver with Master: e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271)
I20250622 13:51:24.505048 974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43754
I20250622 13:51:24.505542 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250622 13:51:24.507172 1113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.507512 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:51:24.507612 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "72d517046abc4f898ddb321158a611d3"
format_stamp: "Formatted at 2025-06-22 13:51:24 on dist-test-slave-tmv0"
I20250622 13:51:24.507732 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:24.517161 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:24.517831 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:24.518402 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:24.519222 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:24.519310 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.519407 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:24.519477 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.534591 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:44537
I20250622 13:51:24.534636 1175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:44537 every 8 connection(s)
I20250622 13:51:24.536319 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:24.539661 1180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:24.540819 1179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:24.541385 413 server_base.cc:1048] running on GCE node
W20250622 13:51:24.542600 1182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:24.543022 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:24.543112 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:24.543193 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600284543158 us; error 0 us; skew 500 ppm
I20250622 13:51:24.543376 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:24.544638 1176 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35365
I20250622 13:51:24.544826 1176 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:24.544950 413 webserver.cc:469] Webserver started at http://127.0.103.67:42691/ using document root <none> and password file <none>
I20250622 13:51:24.545213 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:24.545277 1176 heartbeater.cc:507] Master 127.0.103.126:35365 requested a full tablet report, sending...
I20250622 13:51:24.545333 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:24.545485 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:24.546263 974 ts_manager.cc:194] Registered new tserver with Master: 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537)
I20250622 13:51:24.546219 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
format_stamp: "Formatted at 2025-06-22 13:51:24 on dist-test-slave-tmv0"
I20250622 13:51:24.547287 974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43762
I20250622 13:51:24.549166 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250622 13:51:24.550737 1187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.551069 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250622 13:51:24.551182 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
format_stamp: "Formatted at 2025-06-22 13:51:24 on dist-test-slave-tmv0"
I20250622 13:51:24.551312 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:24.563024 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:24.563614 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:24.564185 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:24.564961 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:24.565032 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.565141 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:24.565199 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:24.579439 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:39473
I20250622 13:51:24.579932 1249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:39473 every 8 connection(s)
I20250622 13:51:24.586108 1250 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35365
I20250622 13:51:24.586244 1250 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:24.586630 1250 heartbeater.cc:507] Master 127.0.103.126:35365 requested a full tablet report, sending...
I20250622 13:51:24.587460 974 ts_manager.cc:194] Registered new tserver with Master: 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:24.588155 974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43770
I20250622 13:51:24.588336 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00785545s
I20250622 13:51:25.506330 1101 heartbeater.cc:499] Master 127.0.103.126:35365 was elected leader, sending a full tablet report...
I20250622 13:51:25.548722 1176 heartbeater.cc:499] Master 127.0.103.126:35365 was elected leader, sending a full tablet report...
I20250622 13:51:25.589568 1250 heartbeater.cc:499] Master 127.0.103.126:35365 was elected leader, sending a full tablet report...
I20250622 13:51:25.616214 413 test_util.cc:276] Using random seed: 1680537661
I20250622 13:51:25.630579 974 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43778:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:51:25.631363 974 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:51:25.653187 1141 tablet_service.cc:1468] Processing CreateTablet for tablet e0764ec090e04b3cb462d33e6e0a95fe (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250622 13:51:25.653270 1140 tablet_service.cc:1468] Processing CreateTablet for tablet e377e7e836f1484a8bd2f462644c4767 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250622 13:51:25.653740 1138 tablet_service.cc:1468] Processing CreateTablet for tablet 0d87f1c84f02409fb7f2c1e8c0270234 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250622 13:51:25.654053 1140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e377e7e836f1484a8bd2f462644c4767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.654389 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d87f1c84f02409fb7f2c1e8c0270234. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.658428 1139 tablet_service.cc:1468] Processing CreateTablet for tablet e774fd12bff54d49b10740499f5e86d5 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250622 13:51:25.659160 1212 tablet_service.cc:1468] Processing CreateTablet for tablet 0d87f1c84f02409fb7f2c1e8c0270234 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250622 13:51:25.659624 1212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d87f1c84f02409fb7f2c1e8c0270234. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.658448 1137 tablet_service.cc:1468] Processing CreateTablet for tablet ad5e40f3433e40999825f73f73f4d5fb (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250622 13:51:25.660867 1214 tablet_service.cc:1468] Processing CreateTablet for tablet e377e7e836f1484a8bd2f462644c4767 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250622 13:51:25.661367 1214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e377e7e836f1484a8bd2f462644c4767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.660019 1211 tablet_service.cc:1468] Processing CreateTablet for tablet ad5e40f3433e40999825f73f73f4d5fb (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250622 13:51:25.664096 1141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0764ec090e04b3cb462d33e6e0a95fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.664420 1213 tablet_service.cc:1468] Processing CreateTablet for tablet e774fd12bff54d49b10740499f5e86d5 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250622 13:51:25.664891 1213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e774fd12bff54d49b10740499f5e86d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.663609 1136 tablet_service.cc:1468] Processing CreateTablet for tablet db3c8931bb80491ca282ae5e2d2e5418 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250622 13:51:25.669953 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad5e40f3433e40999825f73f73f4d5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.671401 1210 tablet_service.cc:1468] Processing CreateTablet for tablet db3c8931bb80491ca282ae5e2d2e5418 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250622 13:51:25.671485 1215 tablet_service.cc:1468] Processing CreateTablet for tablet e0764ec090e04b3cb462d33e6e0a95fe (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250622 13:51:25.672818 1211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad5e40f3433e40999825f73f73f4d5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.672907 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3c8931bb80491ca282ae5e2d2e5418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.673470 1209 tablet_service.cc:1468] Processing CreateTablet for tablet 9da2656343744cfebe86fefdaf1f0c49 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250622 13:51:25.672747 1135 tablet_service.cc:1468] Processing CreateTablet for tablet 9da2656343744cfebe86fefdaf1f0c49 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250622 13:51:25.674964 1134 tablet_service.cc:1468] Processing CreateTablet for tablet 8f78343995bd4bf58432e3a0af6ed595 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250622 13:51:25.675752 1208 tablet_service.cc:1468] Processing CreateTablet for tablet 8f78343995bd4bf58432e3a0af6ed595 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250622 13:51:25.676944 1215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0764ec090e04b3cb462d33e6e0a95fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.677223 1208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f78343995bd4bf58432e3a0af6ed595. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.677457 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e774fd12bff54d49b10740499f5e86d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.677799 1210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3c8931bb80491ca282ae5e2d2e5418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.677909 1135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da2656343744cfebe86fefdaf1f0c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.681114 1134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f78343995bd4bf58432e3a0af6ed595. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.682132 1209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da2656343744cfebe86fefdaf1f0c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.725217 1062 tablet_service.cc:1468] Processing CreateTablet for tablet ad5e40f3433e40999825f73f73f4d5fb (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250622 13:51:25.725988 1062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad5e40f3433e40999825f73f73f4d5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.734632 1066 tablet_service.cc:1468] Processing CreateTablet for tablet e0764ec090e04b3cb462d33e6e0a95fe (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250622 13:51:25.737011 1065 tablet_service.cc:1468] Processing CreateTablet for tablet e377e7e836f1484a8bd2f462644c4767 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250622 13:51:25.739948 1063 tablet_service.cc:1468] Processing CreateTablet for tablet 0d87f1c84f02409fb7f2c1e8c0270234 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250622 13:51:25.740262 1064 tablet_service.cc:1468] Processing CreateTablet for tablet e774fd12bff54d49b10740499f5e86d5 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250622 13:51:25.743388 1061 tablet_service.cc:1468] Processing CreateTablet for tablet db3c8931bb80491ca282ae5e2d2e5418 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250622 13:51:25.743726 1271 tablet_bootstrap.cc:492] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2: Bootstrap starting.
I20250622 13:51:25.744400 1065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e377e7e836f1484a8bd2f462644c4767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.743845 1059 tablet_service.cc:1468] Processing CreateTablet for tablet 8f78343995bd4bf58432e3a0af6ed595 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250622 13:51:25.745114 1064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e774fd12bff54d49b10740499f5e86d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.745211 1060 tablet_service.cc:1468] Processing CreateTablet for tablet 9da2656343744cfebe86fefdaf1f0c49 (DEFAULT_TABLE table=test-workload [id=dcca1494a65049e2bdea4c5c0c299fb1]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250622 13:51:25.745411 1061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3c8931bb80491ca282ae5e2d2e5418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.746407 1066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0764ec090e04b3cb462d33e6e0a95fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.747387 1063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d87f1c84f02409fb7f2c1e8c0270234. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.747632 1270 tablet_bootstrap.cc:492] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3: Bootstrap starting.
I20250622 13:51:25.747817 1059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f78343995bd4bf58432e3a0af6ed595. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.749814 1060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da2656343744cfebe86fefdaf1f0c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:25.779806 1270 tablet_bootstrap.cc:654] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.780128 1272 tablet_bootstrap.cc:492] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925: Bootstrap starting.
I20250622 13:51:25.780671 1271 tablet_bootstrap.cc:654] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.782305 1270 tablet_bootstrap.cc:492] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3: No bootstrap required, opened a new log
I20250622 13:51:25.782824 1272 tablet_bootstrap.cc:654] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.782829 1270 ts_tablet_manager.cc:1397] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.002s
I20250622 13:51:25.782982 1271 tablet_bootstrap.cc:492] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2: No bootstrap required, opened a new log
I20250622 13:51:25.783139 1271 ts_tablet_manager.cc:1397] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.003s
I20250622 13:51:25.783782 1271 raft_consensus.cc:357] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.783815 1270 raft_consensus.cc:357] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.784065 1270 raft_consensus.cc:383] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.784062 1271 raft_consensus.cc:383] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.784181 1270 raft_consensus.cc:738] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.784253 1271 raft_consensus.cc:738] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.784437 1270 consensus_queue.cc:260] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.784544 1271 consensus_queue.cc:260] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.785257 1272 tablet_bootstrap.cc:492] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925: No bootstrap required, opened a new log
I20250622 13:51:25.785342 1271 ts_tablet_manager.cc:1428] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:51:25.785457 1272 ts_tablet_manager.cc:1397] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20250622 13:51:25.785974 1271 tablet_bootstrap.cc:492] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2: Bootstrap starting.
I20250622 13:51:25.786448 1272 raft_consensus.cc:357] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.786686 1272 raft_consensus.cc:383] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.786811 1272 raft_consensus.cc:738] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.787179 1272 consensus_queue.cc:260] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.787714 1271 tablet_bootstrap.cc:654] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.791643 1270 ts_tablet_manager.cc:1428] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20250622 13:51:25.792009 1270 tablet_bootstrap.cc:492] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3: Bootstrap starting.
I20250622 13:51:25.794704 1270 tablet_bootstrap.cc:654] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.796844 1272 ts_tablet_manager.cc:1428] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20250622 13:51:25.797096 1271 tablet_bootstrap.cc:492] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2: No bootstrap required, opened a new log
I20250622 13:51:25.797124 1272 tablet_bootstrap.cc:492] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925: Bootstrap starting.
I20250622 13:51:25.797276 1271 ts_tablet_manager.cc:1397] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20250622 13:51:25.798220 1271 raft_consensus.cc:357] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.798446 1271 raft_consensus.cc:383] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.798554 1271 raft_consensus.cc:738] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.798836 1271 consensus_queue.cc:260] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.799312 1271 ts_tablet_manager.cc:1428] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:51:25.799512 1272 tablet_bootstrap.cc:654] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.800334 1271 tablet_bootstrap.cc:492] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2: Bootstrap starting.
I20250622 13:51:25.802145 1271 tablet_bootstrap.cc:654] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.804584 1270 tablet_bootstrap.cc:492] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3: No bootstrap required, opened a new log
I20250622 13:51:25.804800 1270 ts_tablet_manager.cc:1397] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20250622 13:51:25.805907 1270 raft_consensus.cc:357] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.806185 1270 raft_consensus.cc:383] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.806306 1270 raft_consensus.cc:738] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.806409 1271 tablet_bootstrap.cc:492] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2: No bootstrap required, opened a new log
I20250622 13:51:25.806562 1271 ts_tablet_manager.cc:1397] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250622 13:51:25.806661 1270 consensus_queue.cc:260] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.807320 1270 ts_tablet_manager.cc:1428] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:25.807366 1271 raft_consensus.cc:357] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.807544 1271 raft_consensus.cc:383] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.807639 1270 tablet_bootstrap.cc:492] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3: Bootstrap starting.
I20250622 13:51:25.807674 1272 tablet_bootstrap.cc:492] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925: No bootstrap required, opened a new log
I20250622 13:51:25.807667 1271 raft_consensus.cc:738] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.807845 1272 ts_tablet_manager.cc:1397] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250622 13:51:25.808017 1271 consensus_queue.cc:260] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.808660 1272 raft_consensus.cc:357] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.808853 1272 raft_consensus.cc:383] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.808928 1272 raft_consensus.cc:738] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.809197 1272 consensus_queue.cc:260] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.809726 1272 ts_tablet_manager.cc:1428] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:51:25.810055 1272 tablet_bootstrap.cc:492] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925: Bootstrap starting.
I20250622 13:51:25.810178 1270 tablet_bootstrap.cc:654] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.811748 1271 ts_tablet_manager.cc:1428] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250622 13:51:25.812106 1272 tablet_bootstrap.cc:654] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.813875 1271 tablet_bootstrap.cc:492] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2: Bootstrap starting.
I20250622 13:51:25.815281 1272 tablet_bootstrap.cc:492] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925: No bootstrap required, opened a new log
I20250622 13:51:25.815315 1271 tablet_bootstrap.cc:654] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.815483 1272 ts_tablet_manager.cc:1397] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20250622 13:51:25.816479 1272 raft_consensus.cc:357] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.816720 1272 raft_consensus.cc:383] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.816846 1272 raft_consensus.cc:738] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.817086 1271 tablet_bootstrap.cc:492] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2: No bootstrap required, opened a new log
I20250622 13:51:25.817274 1271 ts_tablet_manager.cc:1397] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250622 13:51:25.817209 1272 consensus_queue.cc:260] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.817751 1272 ts_tablet_manager.cc:1428] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:25.818038 1272 tablet_bootstrap.cc:492] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925: Bootstrap starting.
I20250622 13:51:25.818225 1270 tablet_bootstrap.cc:492] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3: No bootstrap required, opened a new log
I20250622 13:51:25.818179 1271 raft_consensus.cc:357] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.818364 1270 ts_tablet_manager.cc:1397] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20250622 13:51:25.818368 1271 raft_consensus.cc:383] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.818492 1271 raft_consensus.cc:738] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.818749 1271 consensus_queue.cc:260] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.819131 1270 raft_consensus.cc:357] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.819299 1270 raft_consensus.cc:383] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.819392 1270 raft_consensus.cc:738] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.819738 1272 tablet_bootstrap.cc:654] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.819711 1270 consensus_queue.cc:260] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.820060 1271 ts_tablet_manager.cc:1428] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250622 13:51:25.820356 1271 tablet_bootstrap.cc:492] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2: Bootstrap starting.
I20250622 13:51:25.822116 1270 ts_tablet_manager.cc:1428] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250622 13:51:25.822474 1270 tablet_bootstrap.cc:492] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3: Bootstrap starting.
I20250622 13:51:25.822685 1271 tablet_bootstrap.cc:654] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.824618 1270 tablet_bootstrap.cc:654] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.829504 1271 tablet_bootstrap.cc:492] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2: No bootstrap required, opened a new log
I20250622 13:51:25.829650 1271 ts_tablet_manager.cc:1397] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20250622 13:51:25.829892 1272 tablet_bootstrap.cc:492] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925: No bootstrap required, opened a new log
I20250622 13:51:25.830132 1272 ts_tablet_manager.cc:1397] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20250622 13:51:25.830400 1271 raft_consensus.cc:357] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.830571 1271 raft_consensus.cc:383] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.830680 1271 raft_consensus.cc:738] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.830973 1271 consensus_queue.cc:260] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.831166 1272 raft_consensus.cc:357] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.831418 1272 raft_consensus.cc:383] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.831514 1272 raft_consensus.cc:738] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.831817 1272 consensus_queue.cc:260] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.831979 1271 ts_tablet_manager.cc:1428] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:51:25.832270 1271 tablet_bootstrap.cc:492] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2: Bootstrap starting.
I20250622 13:51:25.832329 1272 ts_tablet_manager.cc:1428] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:51:25.832650 1272 tablet_bootstrap.cc:492] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925: Bootstrap starting.
I20250622 13:51:25.834131 1271 tablet_bootstrap.cc:654] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.834470 1272 tablet_bootstrap.cc:654] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.837292 1270 tablet_bootstrap.cc:492] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3: No bootstrap required, opened a new log
I20250622 13:51:25.837524 1270 ts_tablet_manager.cc:1397] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20250622 13:51:25.838526 1270 raft_consensus.cc:357] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.838796 1270 raft_consensus.cc:383] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.838901 1270 raft_consensus.cc:738] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.839201 1270 consensus_queue.cc:260] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.839645 1270 ts_tablet_manager.cc:1428] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:51:25.840016 1270 tablet_bootstrap.cc:492] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3: Bootstrap starting.
I20250622 13:51:25.840305 1272 tablet_bootstrap.cc:492] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925: No bootstrap required, opened a new log
I20250622 13:51:25.840441 1272 ts_tablet_manager.cc:1397] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250622 13:51:25.841236 1272 raft_consensus.cc:357] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.841495 1272 raft_consensus.cc:383] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.841601 1272 raft_consensus.cc:738] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.841843 1270 tablet_bootstrap.cc:654] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.841871 1272 consensus_queue.cc:260] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.842406 1272 ts_tablet_manager.cc:1428] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:51:25.842686 1272 tablet_bootstrap.cc:492] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925: Bootstrap starting.
I20250622 13:51:25.845422 1272 tablet_bootstrap.cc:654] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.845697 1271 tablet_bootstrap.cc:492] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2: No bootstrap required, opened a new log
I20250622 13:51:25.845875 1271 ts_tablet_manager.cc:1397] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20250622 13:51:25.846642 1271 raft_consensus.cc:357] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.846841 1271 raft_consensus.cc:383] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.846927 1271 raft_consensus.cc:738] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.847213 1271 consensus_queue.cc:260] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.847759 1271 ts_tablet_manager.cc:1428] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:51:25.848062 1271 tablet_bootstrap.cc:492] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2: Bootstrap starting.
I20250622 13:51:25.849850 1271 tablet_bootstrap.cc:654] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.851203 1270 tablet_bootstrap.cc:492] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3: No bootstrap required, opened a new log
I20250622 13:51:25.851393 1270 ts_tablet_manager.cc:1397] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250622 13:51:25.852332 1270 raft_consensus.cc:357] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:25.852591 1270 raft_consensus.cc:383] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.852715 1270 raft_consensus.cc:738] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.852953 1270 consensus_queue.cc:260] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:25.853474 1270 ts_tablet_manager.cc:1428] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:25.853873 1270 tablet_bootstrap.cc:492] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3: Bootstrap starting.
I20250622 13:51:25.853874 1272 tablet_bootstrap.cc:492] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925: No bootstrap required, opened a new log
I20250622 13:51:25.854076 1272 ts_tablet_manager.cc:1397] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.010s
I20250622 13:51:25.854959 1272 raft_consensus.cc:357] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.855180 1272 raft_consensus.cc:383] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.855284 1272 raft_consensus.cc:738] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.855502 1271 tablet_bootstrap.cc:492] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2: No bootstrap required, opened a new log
I20250622 13:51:25.855650 1271 ts_tablet_manager.cc:1397] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250622 13:51:25.855588 1272 consensus_queue.cc:260] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.855793 1270 tablet_bootstrap.cc:654] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.857064 1271 raft_consensus.cc:357] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:25.857250 1271 raft_consensus.cc:383] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.857383 1271 raft_consensus.cc:738] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.857633 1271 consensus_queue.cc:260] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:25.856511 1272 ts_tablet_manager.cc:1428] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250622 13:51:25.858871 1272 tablet_bootstrap.cc:492] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925: Bootstrap starting.
I20250622 13:51:25.860103 1271 ts_tablet_manager.cc:1428] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250622 13:51:25.861086 1271 tablet_bootstrap.cc:492] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2: Bootstrap starting.
I20250622 13:51:25.861254 1272 tablet_bootstrap.cc:654] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.862761 1271 tablet_bootstrap.cc:654] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.865394 1271 tablet_bootstrap.cc:492] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2: No bootstrap required, opened a new log
I20250622 13:51:25.865588 1271 ts_tablet_manager.cc:1397] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250622 13:51:25.865681 1270 tablet_bootstrap.cc:492] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3: No bootstrap required, opened a new log
I20250622 13:51:25.866333 1270 ts_tablet_manager.cc:1397] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20250622 13:51:25.866648 1272 tablet_bootstrap.cc:492] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925: No bootstrap required, opened a new log
I20250622 13:51:25.866786 1272 ts_tablet_manager.cc:1397] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20250622 13:51:25.867002 1270 raft_consensus.cc:357] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.866906 1271 raft_consensus.cc:357] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.867190 1270 raft_consensus.cc:383] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.867236 1271 raft_consensus.cc:383] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.867314 1270 raft_consensus.cc:738] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.867317 1271 raft_consensus.cc:738] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.867556 1272 raft_consensus.cc:357] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:25.867594 1271 consensus_queue.cc:260] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.867635 1270 consensus_queue.cc:260] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.867707 1272 raft_consensus.cc:383] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.868139 1270 ts_tablet_manager.cc:1428] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:51:25.868140 1271 ts_tablet_manager.cc:1428] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:51:25.868461 1270 tablet_bootstrap.cc:492] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3: Bootstrap starting.
I20250622 13:51:25.868490 1272 raft_consensus.cc:738] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.868757 1272 consensus_queue.cc:260] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:25.869302 1272 ts_tablet_manager.cc:1428] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:51:25.870464 1270 tablet_bootstrap.cc:654] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.870496 1272 tablet_bootstrap.cc:492] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925: Bootstrap starting.
I20250622 13:51:25.872237 1272 tablet_bootstrap.cc:654] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.877718 1270 tablet_bootstrap.cc:492] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3: No bootstrap required, opened a new log
I20250622 13:51:25.877796 1272 tablet_bootstrap.cc:492] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925: No bootstrap required, opened a new log
I20250622 13:51:25.877913 1270 ts_tablet_manager.cc:1397] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20250622 13:51:25.877990 1272 ts_tablet_manager.cc:1397] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20250622 13:51:25.878639 1270 raft_consensus.cc:357] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.878794 1270 raft_consensus.cc:383] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.878717 1272 raft_consensus.cc:357] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.878863 1270 raft_consensus.cc:738] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.878921 1272 raft_consensus.cc:383] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.879010 1272 raft_consensus.cc:738] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.879057 1270 consensus_queue.cc:260] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.879251 1272 consensus_queue.cc:260] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.879498 1270 ts_tablet_manager.cc:1428] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:51:25.879748 1272 ts_tablet_manager.cc:1428] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:51:25.879783 1270 tablet_bootstrap.cc:492] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3: Bootstrap starting.
I20250622 13:51:25.881294 1270 tablet_bootstrap.cc:654] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:25.883438 1270 tablet_bootstrap.cc:492] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3: No bootstrap required, opened a new log
I20250622 13:51:25.883617 1270 ts_tablet_manager.cc:1397] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:51:25.884270 1270 raft_consensus.cc:357] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.884429 1270 raft_consensus.cc:383] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:25.884513 1270 raft_consensus.cc:738] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Initialized, Role: FOLLOWER
I20250622 13:51:25.884744 1270 consensus_queue.cc:260] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.885164 1270 ts_tablet_manager.cc:1428] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:51:25.889991 1276 raft_consensus.cc:491] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:25.890147 1276 raft_consensus.cc:513] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.890945 1276 leader_election.cc:290] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:25.896461 1277 raft_consensus.cc:491] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:25.896634 1277 raft_consensus.cc:513] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.897841 1277 leader_election.cc:290] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271), 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537)
I20250622 13:51:25.899341 1076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3c8931bb80491ca282ae5e2d2e5418" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e613b3722f774902b7f3f2569a18c925" is_pre_election: true
I20250622 13:51:25.899703 1076 raft_consensus.cc:2466] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 0.
I20250622 13:51:25.900314 1117 leader_election.cc:304] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:25.900633 1276 raft_consensus.cc:2802] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:25.900758 1276 raft_consensus.cc:491] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:25.900868 1276 raft_consensus.cc:3058] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.905813 1276 raft_consensus.cc:513] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.906574 1277 raft_consensus.cc:491] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:25.906754 1277 raft_consensus.cc:513] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.907078 1076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3c8931bb80491ca282ae5e2d2e5418" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e613b3722f774902b7f3f2569a18c925"
I20250622 13:51:25.907236 1076 raft_consensus.cc:3058] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.907341 1277 leader_election.cc:290] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271), 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537)
I20250622 13:51:25.911276 1076 raft_consensus.cc:2466] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 1.
I20250622 13:51:25.911113 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3c8931bb80491ca282ae5e2d2e5418" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" is_pre_election: true
I20250622 13:51:25.911546 1225 raft_consensus.cc:2466] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 0.
I20250622 13:51:25.911712 1117 leader_election.cc:304] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:25.912720 1276 leader_election.cc:290] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 1 election: Requested vote from peers e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:25.912932 1276 raft_consensus.cc:2802] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:25.913074 1276 raft_consensus.cc:695] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 1 LEADER]: Becoming Leader. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Running, Role: LEADER
I20250622 13:51:25.913302 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3c8931bb80491ca282ae5e2d2e5418" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
I20250622 13:51:25.913403 1276 consensus_queue.cc:237] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.913580 1225 raft_consensus.cc:3058] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.918023 1225 raft_consensus.cc:2466] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 1.
I20250622 13:51:25.921440 1076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f78343995bd4bf58432e3a0af6ed595" candidate_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e613b3722f774902b7f3f2569a18c925" is_pre_election: true
I20250622 13:51:25.922094 1076 raft_consensus.cc:2466] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b582b3d93dc4a7ebc3128b41d1895f2 in term 0.
I20250622 13:51:25.921442 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3c8931bb80491ca282ae5e2d2e5418" candidate_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e613b3722f774902b7f3f2569a18c925" is_pre_election: true
I20250622 13:51:25.923259 1191 leader_election.cc:304] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:25.923303 1075 raft_consensus.cc:2391] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b582b3d93dc4a7ebc3128b41d1895f2 in current term 1: Already voted for candidate 72d517046abc4f898ddb321158a611d3 in this term.
I20250622 13:51:25.924458 1277 raft_consensus.cc:2802] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:25.924787 1277 raft_consensus.cc:491] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:25.926069 1277 raft_consensus.cc:3058] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.925896 973 catalog_manager.cc:5582] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72d517046abc4f898ddb321158a611d3 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "72d517046abc4f898ddb321158a611d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:25.929033 1277 raft_consensus.cc:513] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.929642 1151 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f78343995bd4bf58432e3a0af6ed595" candidate_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3" is_pre_election: true
I20250622 13:51:25.929772 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db3c8931bb80491ca282ae5e2d2e5418" candidate_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3" is_pre_election: true
I20250622 13:51:25.929970 1151 raft_consensus.cc:2466] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b582b3d93dc4a7ebc3128b41d1895f2 in term 0.
I20250622 13:51:25.930058 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f78343995bd4bf58432e3a0af6ed595" candidate_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e613b3722f774902b7f3f2569a18c925"
I20250622 13:51:25.930218 1075 raft_consensus.cc:3058] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.930644 1188 leader_election.cc:304] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2; no voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925
I20250622 13:51:25.931257 1277 leader_election.cc:290] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [CANDIDATE]: Term 1 election: Requested vote from peers e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271), 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537)
I20250622 13:51:25.931424 1277 raft_consensus.cc:2747] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250622 13:51:25.932137 1278 raft_consensus.cc:491] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:25.932281 1278 raft_consensus.cc:513] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.934156 1278 leader_election.cc:290] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:25.934927 1075 raft_consensus.cc:2466] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b582b3d93dc4a7ebc3128b41d1895f2 in term 1.
I20250622 13:51:25.935346 1191 leader_election.cc:304] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:25.937278 1307 raft_consensus.cc:2802] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:25.937451 1307 raft_consensus.cc:695] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 LEADER]: Becoming Leader. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Running, Role: LEADER
I20250622 13:51:25.937716 1307 consensus_queue.cc:237] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.939378 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f78343995bd4bf58432e3a0af6ed595" candidate_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3"
I20250622 13:51:25.939591 1150 raft_consensus.cc:3058] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.940822 973 catalog_manager.cc:5582] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:25.941911 1150 raft_consensus.cc:2466] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b582b3d93dc4a7ebc3128b41d1895f2 in term 1.
I20250622 13:51:25.952450 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e377e7e836f1484a8bd2f462644c4767" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3" is_pre_election: true
I20250622 13:51:25.952729 1150 raft_consensus.cc:2466] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:25.952847 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e377e7e836f1484a8bd2f462644c4767" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" is_pre_election: true
I20250622 13:51:25.953058 1225 raft_consensus.cc:2466] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:25.953136 1039 leader_election.cc:304] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:25.953495 1278 raft_consensus.cc:2802] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:25.953655 1278 raft_consensus.cc:491] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:25.953769 1278 raft_consensus.cc:3058] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.955783 1278 raft_consensus.cc:513] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.956398 1278 leader_election.cc:290] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Requested vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:25.956853 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e377e7e836f1484a8bd2f462644c4767" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3"
I20250622 13:51:25.956883 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e377e7e836f1484a8bd2f462644c4767" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
I20250622 13:51:25.957016 1150 raft_consensus.cc:3058] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.957072 1225 raft_consensus.cc:3058] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.959101 1150 raft_consensus.cc:2466] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:25.959244 1225 raft_consensus.cc:2466] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:25.959487 1039 leader_election.cc:304] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:25.959760 1278 raft_consensus.cc:2802] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:25.960165 1278 raft_consensus.cc:695] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Becoming Leader. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Running, Role: LEADER
I20250622 13:51:25.960489 1278 consensus_queue.cc:237] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.963194 973 catalog_manager.cc:5582] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to e613b3722f774902b7f3f2569a18c925 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e613b3722f774902b7f3f2569a18c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:25.993345 1278 raft_consensus.cc:491] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:25.993611 1278 raft_consensus.cc:513] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.994285 1278 leader_election.cc:290] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:25.994881 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e774fd12bff54d49b10740499f5e86d5" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3" is_pre_election: true
I20250622 13:51:25.994891 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e774fd12bff54d49b10740499f5e86d5" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" is_pre_election: true
I20250622 13:51:25.995107 1150 raft_consensus.cc:2466] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:25.995692 1225 raft_consensus.cc:2466] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:25.995779 1039 leader_election.cc:304] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:25.996112 1278 raft_consensus.cc:2802] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:25.996220 1278 raft_consensus.cc:491] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:25.996327 1278 raft_consensus.cc:3058] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.998457 1278 raft_consensus.cc:513] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:25.999229 1278 leader_election.cc:290] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Requested vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:25.999563 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e774fd12bff54d49b10740499f5e86d5" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3"
I20250622 13:51:25.999627 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e774fd12bff54d49b10740499f5e86d5" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
I20250622 13:51:25.999763 1150 raft_consensus.cc:3058] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:25.999788 1225 raft_consensus.cc:3058] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.001793 1225 raft_consensus.cc:2466] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:26.001866 1150 raft_consensus.cc:2466] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:26.002213 1041 leader_election.cc:304] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:26.002465 1278 raft_consensus.cc:2802] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:26.002651 1278 raft_consensus.cc:695] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Becoming Leader. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Running, Role: LEADER
I20250622 13:51:26.002955 1278 consensus_queue.cc:237] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.005831 973 catalog_manager.cc:5582] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to e613b3722f774902b7f3f2569a18c925 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e613b3722f774902b7f3f2569a18c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:26.018859 1278 raft_consensus.cc:491] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:26.019076 1278 raft_consensus.cc:513] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.019752 1278 leader_election.cc:290] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.020238 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad5e40f3433e40999825f73f73f4d5fb" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" is_pre_election: true
I20250622 13:51:26.020243 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad5e40f3433e40999825f73f73f4d5fb" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3" is_pre_election: true
I20250622 13:51:26.020509 1150 raft_consensus.cc:2466] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:26.020510 1225 raft_consensus.cc:2466] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:26.020908 1041 leader_election.cc:304] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:26.021133 1278 raft_consensus.cc:2802] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:26.021243 1278 raft_consensus.cc:491] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:26.021314 1278 raft_consensus.cc:3058] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.023180 1278 raft_consensus.cc:513] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.023694 1278 leader_election.cc:290] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Requested vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.024098 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad5e40f3433e40999825f73f73f4d5fb" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3"
I20250622 13:51:26.024160 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad5e40f3433e40999825f73f73f4d5fb" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
I20250622 13:51:26.024264 1150 raft_consensus.cc:3058] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.024845 1225 raft_consensus.cc:3058] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.026517 1150 raft_consensus.cc:2466] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:26.026806 1225 raft_consensus.cc:2466] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:26.026925 1039 leader_election.cc:304] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:26.027215 1278 raft_consensus.cc:2802] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:26.027344 1278 raft_consensus.cc:695] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Becoming Leader. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Running, Role: LEADER
I20250622 13:51:26.027634 1278 consensus_queue.cc:237] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.030157 973 catalog_manager.cc:5582] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to e613b3722f774902b7f3f2569a18c925 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e613b3722f774902b7f3f2569a18c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:26.044726 1278 raft_consensus.cc:491] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:26.044952 1278 raft_consensus.cc:513] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:26.045636 1278 leader_election.cc:290] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.046136 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0764ec090e04b3cb462d33e6e0a95fe" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3" is_pre_election: true
I20250622 13:51:26.046214 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0764ec090e04b3cb462d33e6e0a95fe" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" is_pre_election: true
I20250622 13:51:26.046334 1150 raft_consensus.cc:2466] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:26.046443 1225 raft_consensus.cc:2466] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:26.046730 1039 leader_election.cc:304] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:26.047004 1278 raft_consensus.cc:2802] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:26.047091 1278 raft_consensus.cc:491] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:26.047199 1278 raft_consensus.cc:3058] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.049043 1278 raft_consensus.cc:513] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:26.049741 1278 leader_election.cc:290] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Requested vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.050175 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0764ec090e04b3cb462d33e6e0a95fe" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3"
I20250622 13:51:26.050184 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0764ec090e04b3cb462d33e6e0a95fe" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
I20250622 13:51:26.050396 1225 raft_consensus.cc:3058] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.050403 1150 raft_consensus.cc:3058] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.053236 1150 raft_consensus.cc:2466] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:26.053242 1225 raft_consensus.cc:2466] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:26.053722 1041 leader_election.cc:304] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:26.054005 1278 raft_consensus.cc:2802] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:26.054167 1278 raft_consensus.cc:695] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Becoming Leader. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Running, Role: LEADER
I20250622 13:51:26.054442 1278 consensus_queue.cc:237] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:26.056902 973 catalog_manager.cc:5582] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to e613b3722f774902b7f3f2569a18c925 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e613b3722f774902b7f3f2569a18c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:26.109526 1305 raft_consensus.cc:491] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:26.109750 1305 raft_consensus.cc:513] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.110502 1305 leader_election.cc:290] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.111032 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9da2656343744cfebe86fefdaf1f0c49" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" is_pre_election: true
I20250622 13:51:26.111058 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9da2656343744cfebe86fefdaf1f0c49" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e613b3722f774902b7f3f2569a18c925" is_pre_election: true
I20250622 13:51:26.111349 1225 raft_consensus.cc:2466] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 0.
I20250622 13:51:26.111424 1075 raft_consensus.cc:2466] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 0.
I20250622 13:51:26.111749 1116 leader_election.cc:304] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, 72d517046abc4f898ddb321158a611d3; no voters:
I20250622 13:51:26.112066 1305 raft_consensus.cc:2802] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:26.112181 1305 raft_consensus.cc:491] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:26.112280 1305 raft_consensus.cc:3058] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.114750 1305 raft_consensus.cc:513] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.115288 1305 leader_election.cc:290] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 1 election: Requested vote from peers e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.115772 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9da2656343744cfebe86fefdaf1f0c49" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
I20250622 13:51:26.115772 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9da2656343744cfebe86fefdaf1f0c49" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e613b3722f774902b7f3f2569a18c925"
I20250622 13:51:26.116021 1075 raft_consensus.cc:3058] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.116099 1225 raft_consensus.cc:3058] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.118902 1225 raft_consensus.cc:2466] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 1.
I20250622 13:51:26.118903 1075 raft_consensus.cc:2466] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 1.
I20250622 13:51:26.119359 1117 leader_election.cc:304] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:26.119689 1305 raft_consensus.cc:2802] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:26.119844 1305 raft_consensus.cc:695] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 1 LEADER]: Becoming Leader. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Running, Role: LEADER
I20250622 13:51:26.120149 1305 consensus_queue.cc:237] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.123045 973 catalog_manager.cc:5582] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72d517046abc4f898ddb321158a611d3 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "72d517046abc4f898ddb321158a611d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:26.123842 1278 raft_consensus.cc:491] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:26.124014 1278 raft_consensus.cc:513] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.124763 1278 leader_election.cc:290] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.125494 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d87f1c84f02409fb7f2c1e8c0270234" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" is_pre_election: true
I20250622 13:51:26.125682 1225 raft_consensus.cc:2466] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:26.125057 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d87f1c84f02409fb7f2c1e8c0270234" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3" is_pre_election: true
I20250622 13:51:26.126183 1041 leader_election.cc:304] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:26.127246 1150 raft_consensus.cc:2466] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 0.
I20250622 13:51:26.127354 1278 raft_consensus.cc:2802] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:26.127485 1278 raft_consensus.cc:491] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:26.127604 1278 raft_consensus.cc:3058] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.129432 1278 raft_consensus.cc:513] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.129942 1278 leader_election.cc:290] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Requested vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.130316 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d87f1c84f02409fb7f2c1e8c0270234" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d517046abc4f898ddb321158a611d3"
I20250622 13:51:26.130373 1225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d87f1c84f02409fb7f2c1e8c0270234" candidate_uuid: "e613b3722f774902b7f3f2569a18c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
I20250622 13:51:26.130476 1150 raft_consensus.cc:3058] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.130511 1225 raft_consensus.cc:3058] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:26.132309 1150 raft_consensus.cc:2466] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:26.132331 1225 raft_consensus.cc:2466] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e613b3722f774902b7f3f2569a18c925 in term 1.
I20250622 13:51:26.132656 1039 leader_election.cc:304] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d517046abc4f898ddb321158a611d3, e613b3722f774902b7f3f2569a18c925; no voters:
I20250622 13:51:26.132887 1278 raft_consensus.cc:2802] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:26.133016 1278 raft_consensus.cc:695] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Becoming Leader. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Running, Role: LEADER
I20250622 13:51:26.133249 1278 consensus_queue.cc:237] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.135825 973 catalog_manager.cc:5582] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to e613b3722f774902b7f3f2569a18c925 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e613b3722f774902b7f3f2569a18c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:26.236102 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:26.238953 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:26.239117 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:26.239730 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:26.239768 413 server_base.cc:1048] running on GCE node
I20250622 13:51:26.240093 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:26.240167 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:26.240207 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600286240189 us; error 0 us; skew 500 ppm
I20250622 13:51:26.240361 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:26.241793 413 webserver.cc:469] Webserver started at http://127.0.103.68:43137/ using document root <none> and password file <none>
I20250622 13:51:26.242031 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:26.242105 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:26.242272 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:26.242964 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-3-root/instance:
uuid: "15262ae29db04ae59ac63f8e50ce6009"
format_stamp: "Formatted at 2025-06-22 13:51:26 on dist-test-slave-tmv0"
I20250622 13:51:26.245258 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:26.246876 1326 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:26.247260 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:51:26.247354 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-3-root
uuid: "15262ae29db04ae59ac63f8e50ce6009"
format_stamp: "Formatted at 2025-06-22 13:51:26 on dist-test-slave-tmv0"
I20250622 13:51:26.247470 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750600281110842-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:26.265090 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:26.265705 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:26.266291 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:26.267148 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:26.267228 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:26.267325 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:26.267410 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:26.285080 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:35581
I20250622 13:51:26.285140 1388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:35581 every 8 connection(s)
I20250622 13:51:26.292765 1389 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35365
I20250622 13:51:26.292912 1389 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:26.293331 1389 heartbeater.cc:507] Master 127.0.103.126:35365 requested a full tablet report, sending...
I20250622 13:51:26.294219 973 ts_manager.cc:194] Registered new tserver with Master: 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68:35581)
I20250622 13:51:26.295017 973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43782
I20250622 13:51:26.307314 1305 consensus_queue.cc:1035] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.312350 1305 consensus_queue.cc:1035] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.392937 1277 consensus_queue.cc:1035] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.396940 1277 consensus_queue.cc:1035] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.407996 1027 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:26.409564 1278 consensus_queue.cc:1035] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.416625 1403 consensus_queue.cc:1035] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.418535 1075 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ad5e40f3433e40999825f73f73f4d5fb"
dest_uuid: "e613b3722f774902b7f3f2569a18c925"
mode: GRACEFUL
new_leader_uuid: "72d517046abc4f898ddb321158a611d3"
from {username='slave'} at 127.0.0.1:48100
I20250622 13:51:26.418761 1075 raft_consensus.cc:604] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Received request to transfer leadership to TS 72d517046abc4f898ddb321158a611d3
I20250622 13:51:26.420091 1075 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e0764ec090e04b3cb462d33e6e0a95fe"
dest_uuid: "e613b3722f774902b7f3f2569a18c925"
mode: GRACEFUL
new_leader_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
from {username='slave'} at 127.0.0.1:48100
I20250622 13:51:26.420274 1075 raft_consensus.cc:604] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Received request to transfer leadership to TS 3b582b3d93dc4a7ebc3128b41d1895f2
I20250622 13:51:26.420606 1027 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250622 13:51:26.420871 1027 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:26.488255 1403 consensus_queue.cc:1035] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.492359 1311 consensus_queue.cc:1035] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.494829 1311 raft_consensus.cc:991] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925: : Instructing follower 3b582b3d93dc4a7ebc3128b41d1895f2 to start an election
I20250622 13:51:26.495019 1311 raft_consensus.cc:1079] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Signalling peer 3b582b3d93dc4a7ebc3128b41d1895f2 to start an election
I20250622 13:51:26.495791 1225 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e0764ec090e04b3cb462d33e6e0a95fe"
dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
from {username='slave'} at 127.0.0.1:60198
I20250622 13:51:26.495993 1225 raft_consensus.cc:491] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:51:26.496116 1225 raft_consensus.cc:3058] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:51:26.499071 1225 raft_consensus.cc:513] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:26.499788 1225 leader_election.cc:290] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [CANDIDATE]: Term 2 election: Requested vote from peers 72d517046abc4f898ddb321158a611d3 (127.0.103.66:44537), e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271)
I20250622 13:51:26.500095 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0764ec090e04b3cb462d33e6e0a95fe" candidate_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "72d517046abc4f898ddb321158a611d3"
I20250622 13:51:26.500247 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0764ec090e04b3cb462d33e6e0a95fe" candidate_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e613b3722f774902b7f3f2569a18c925"
I20250622 13:51:26.500314 1150 raft_consensus.cc:3058] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:51:26.500397 1075 raft_consensus.cc:3053] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:51:26.500499 1075 raft_consensus.cc:738] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Running, Role: LEADER
I20250622 13:51:26.500782 1075 consensus_queue.cc:260] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:26.501137 1075 raft_consensus.cc:3058] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:51:26.501675 1311 consensus_queue.cc:1035] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.502696 1150 raft_consensus.cc:2466] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b582b3d93dc4a7ebc3128b41d1895f2 in term 2.
I20250622 13:51:26.503123 1188 leader_election.cc:304] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, 72d517046abc4f898ddb321158a611d3; no voters:
I20250622 13:51:26.503429 1277 raft_consensus.cc:2802] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:51:26.503582 1277 raft_consensus.cc:695] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 LEADER]: Becoming Leader. State: Replica: 3b582b3d93dc4a7ebc3128b41d1895f2, State: Running, Role: LEADER
I20250622 13:51:26.503842 1075 raft_consensus.cc:2466] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b582b3d93dc4a7ebc3128b41d1895f2 in term 2.
I20250622 13:51:26.503890 1277 consensus_queue.cc:237] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } }
I20250622 13:51:26.509665 1311 consensus_queue.cc:1035] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.509968 974 catalog_manager.cc:5582] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 reported cstate change: term changed from 1 to 2, leader changed from e613b3722f774902b7f3f2569a18c925 (127.0.103.65) to 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:26.510807 1403 consensus_queue.cc:1035] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.512492 1311 raft_consensus.cc:991] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925: : Instructing follower 72d517046abc4f898ddb321158a611d3 to start an election
I20250622 13:51:26.512648 1311 raft_consensus.cc:1079] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Signalling peer 72d517046abc4f898ddb321158a611d3 to start an election
I20250622 13:51:26.513410 1151 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ad5e40f3433e40999825f73f73f4d5fb"
dest_uuid: "72d517046abc4f898ddb321158a611d3"
from {username='slave'} at 127.0.0.1:48558
I20250622 13:51:26.513585 1151 raft_consensus.cc:491] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:51:26.513720 1151 raft_consensus.cc:3058] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:51:26.516758 1278 consensus_queue.cc:1035] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.517814 1151 raft_consensus.cc:513] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.519019 1278 consensus_queue.cc:1035] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.519752 1151 leader_election.cc:290] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 2 election: Requested vote from peers e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271), 3b582b3d93dc4a7ebc3128b41d1895f2 (127.0.103.67:39473)
I20250622 13:51:26.520108 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad5e40f3433e40999825f73f73f4d5fb" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e613b3722f774902b7f3f2569a18c925"
I20250622 13:51:26.520298 1075 raft_consensus.cc:3053] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:51:26.520206 1224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad5e40f3433e40999825f73f73f4d5fb" candidate_uuid: "72d517046abc4f898ddb321158a611d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2"
I20250622 13:51:26.520426 1075 raft_consensus.cc:738] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e613b3722f774902b7f3f2569a18c925, State: Running, Role: LEADER
I20250622 13:51:26.520474 1224 raft_consensus.cc:3058] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:51:26.520716 1075 consensus_queue.cc:260] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.521057 1075 raft_consensus.cc:3058] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:51:26.524633 1224 raft_consensus.cc:2466] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 2.
I20250622 13:51:26.525043 1116 leader_election.cc:304] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b582b3d93dc4a7ebc3128b41d1895f2, 72d517046abc4f898ddb321158a611d3; no voters:
I20250622 13:51:26.525213 1075 raft_consensus.cc:2466] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d517046abc4f898ddb321158a611d3 in term 2.
I20250622 13:51:26.525305 1305 raft_consensus.cc:2802] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:51:26.525503 1305 raft_consensus.cc:695] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 2 LEADER]: Becoming Leader. State: Replica: 72d517046abc4f898ddb321158a611d3, State: Running, Role: LEADER
I20250622 13:51:26.525830 1305 consensus_queue.cc:237] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } }
I20250622 13:51:26.532680 1403 consensus_queue.cc:1035] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.534327 974 catalog_manager.cc:5582] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 reported cstate change: term changed from 1 to 2, leader changed from e613b3722f774902b7f3f2569a18c925 (127.0.103.65) to 72d517046abc4f898ddb321158a611d3 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "72d517046abc4f898ddb321158a611d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:26.656582 1305 consensus_queue.cc:1035] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.660807 1276 consensus_queue.cc:1035] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:26.926270 1075 raft_consensus.cc:1273] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Refusing update from remote peer 3b582b3d93dc4a7ebc3128b41d1895f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:51:26.926841 1399 consensus_queue.cc:1035] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:26.929957 1149 raft_consensus.cc:1273] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 2 FOLLOWER]: Refusing update from remote peer 3b582b3d93dc4a7ebc3128b41d1895f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:51:26.930609 1277 consensus_queue.cc:1035] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:26.975245 1075 raft_consensus.cc:1273] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Refusing update from remote peer 72d517046abc4f898ddb321158a611d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:51:26.975832 1415 consensus_queue.cc:1035] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:26.978644 1224 raft_consensus.cc:1273] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Refusing update from remote peer 72d517046abc4f898ddb321158a611d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:51:26.979194 1276 consensus_queue.cc:1035] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:27.296247 1389 heartbeater.cc:499] Master 127.0.103.126:35365 was elected leader, sending a full tablet report...
I20250622 13:51:28.421298 1027 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:28.422681 1027 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:28.422852 1027 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:30.423359 1027 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:30.424659 1027 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:30.424854 1027 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:31.424201 1076 consensus_queue.cc:237] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } }
I20250622 13:51:31.426908 1151 raft_consensus.cc:1273] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Refusing update from remote peer e613b3722f774902b7f3f2569a18c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:31.427632 1224 raft_consensus.cc:1273] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Refusing update from remote peer e613b3722f774902b7f3f2569a18c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:31.427601 1278 consensus_queue.cc:1035] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:31.428247 1278 consensus_queue.cc:1035] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:31.430730 1419 raft_consensus.cc:2953] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.433272 1151 raft_consensus.cc:2953] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
W20250622 13:51:31.436700 1040 consensus_peers.cc:487] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 -> Peer 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68:35581): Couldn't send request to peer 15262ae29db04ae59ac63f8e50ce6009. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e774fd12bff54d49b10740499f5e86d5. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:51:31.436702 1224 raft_consensus.cc:2953] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.442818 974 catalog_manager.cc:5582] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 reported cstate change: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "e613b3722f774902b7f3f2569a18c925" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.450699 1151 consensus_queue.cc:237] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } attrs { replace: true } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } }
I20250622 13:51:31.453774 1076 raft_consensus.cc:1273] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Refusing update from remote peer 72d517046abc4f898ddb321158a611d3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:51:31.454005 1224 raft_consensus.cc:1273] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Refusing update from remote peer 72d517046abc4f898ddb321158a611d3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:51:31.454295 1432 consensus_queue.cc:1035] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:51:31.454679 1276 consensus_queue.cc:1035] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:51:31.457415 1276 raft_consensus.cc:2953] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } attrs { replace: true } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.457931 1076 raft_consensus.cc:2953] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } attrs { replace: true } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.458576 1224 raft_consensus.cc:2953] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } attrs { replace: true } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.463032 972 catalog_manager.cc:5582] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 reported cstate change: config changed from index -1 to 3, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "72d517046abc4f898ddb321158a611d3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250622 13:51:31.465204 1115 consensus_peers.cc:487] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 -> Peer 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68:35581): Couldn't send request to peer 15262ae29db04ae59ac63f8e50ce6009. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad5e40f3433e40999825f73f73f4d5fb. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:51:31.465772 1076 consensus_queue.cc:237] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } attrs { replace: true } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } }
I20250622 13:51:31.470453 1224 raft_consensus.cc:1273] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Refusing update from remote peer e613b3722f774902b7f3f2569a18c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:31.470580 1151 raft_consensus.cc:1273] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Refusing update from remote peer e613b3722f774902b7f3f2569a18c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:31.470894 1419 consensus_queue.cc:1035] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250622 13:51:31.471110 1040 consensus_peers.cc:487] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 -> Peer 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68:35581): Couldn't send request to peer 15262ae29db04ae59ac63f8e50ce6009. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d87f1c84f02409fb7f2c1e8c0270234. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:51:31.472360 1419 consensus_queue.cc:1035] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:31.473191 1424 raft_consensus.cc:2953] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } attrs { replace: true } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.473841 1224 raft_consensus.cc:2953] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } attrs { replace: true } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.478780 974 catalog_manager.cc:5582] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 reported cstate change: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "e613b3722f774902b7f3f2569a18c925" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:31.481628 1151 raft_consensus.cc:2953] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } attrs { replace: true } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.488538 1224 consensus_queue.cc:237] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } }
I20250622 13:51:31.490934 1076 raft_consensus.cc:1273] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Refusing update from remote peer 3b582b3d93dc4a7ebc3128b41d1895f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:31.491494 1307 consensus_queue.cc:1035] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:31.492147 1151 raft_consensus.cc:1273] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Refusing update from remote peer 3b582b3d93dc4a7ebc3128b41d1895f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:31.493129 1307 consensus_queue.cc:1035] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:31.494246 1441 raft_consensus.cc:2953] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.494794 1076 raft_consensus.cc:2953] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.500811 973 catalog_manager.cc:5582] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 reported cstate change: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250622 13:51:31.501116 1189 consensus_peers.cc:487] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 -> Peer 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68:35581): Couldn't send request to peer 15262ae29db04ae59ac63f8e50ce6009. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f78343995bd4bf58432e3a0af6ed595. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:51:31.503512 1151 raft_consensus.cc:2953] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 15262ae29db04ae59ac63f8e50ce6009 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e613b3722f774902b7f3f2569a18c925" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46271 } } peers { permanent_uuid: "72d517046abc4f898ddb321158a611d3" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 44537 } attrs { replace: true } } peers { permanent_uuid: "3b582b3d93dc4a7ebc3128b41d1895f2" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 39473 } } peers { permanent_uuid: "15262ae29db04ae59ac63f8e50ce6009" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35581 } attrs { promote: true } } }
I20250622 13:51:31.543845 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
W20250622 13:51:31.544950 961 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20250622 13:51:31.547607 1022 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250622 13:51:31.548365 1022 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250622 13:51:31.550330 1022 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250622 13:51:31.550454 1022 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250622 13:51:31.553155 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:31.553438 413 tablet_replica.cc:331] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925: stopping tablet replica
I20250622 13:51:31.553623 413 raft_consensus.cc:2241] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.553900 413 raft_consensus.cc:2270] T 8f78343995bd4bf58432e3a0af6ed595 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.554728 413 tablet_replica.cc:331] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925: stopping tablet replica
I20250622 13:51:31.554900 413 raft_consensus.cc:2241] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.555078 413 raft_consensus.cc:2270] T e0764ec090e04b3cb462d33e6e0a95fe P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.555608 413 tablet_replica.cc:331] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925: stopping tablet replica
I20250622 13:51:31.555748 413 raft_consensus.cc:2241] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.556206 413 raft_consensus.cc:2270] T e774fd12bff54d49b10740499f5e86d5 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.556826 413 tablet_replica.cc:331] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925: stopping tablet replica
I20250622 13:51:31.556993 413 raft_consensus.cc:2241] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.557282 413 raft_consensus.cc:2270] T e377e7e836f1484a8bd2f462644c4767 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.557804 413 tablet_replica.cc:331] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925: stopping tablet replica
I20250622 13:51:31.557947 413 raft_consensus.cc:2241] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.558180 413 raft_consensus.cc:2270] T ad5e40f3433e40999825f73f73f4d5fb P e613b3722f774902b7f3f2569a18c925 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.558828 413 tablet_replica.cc:331] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925: stopping tablet replica
I20250622 13:51:31.558976 413 raft_consensus.cc:2241] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.559286 413 raft_consensus.cc:2270] T 0d87f1c84f02409fb7f2c1e8c0270234 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.559932 413 tablet_replica.cc:331] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925: stopping tablet replica
I20250622 13:51:31.560057 413 raft_consensus.cc:2241] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.560220 413 raft_consensus.cc:2270] T db3c8931bb80491ca282ae5e2d2e5418 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.560685 413 tablet_replica.cc:331] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925: stopping tablet replica
I20250622 13:51:31.560859 413 raft_consensus.cc:2241] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.561029 413 raft_consensus.cc:2270] T 9da2656343744cfebe86fefdaf1f0c49 P e613b3722f774902b7f3f2569a18c925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.577586 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:51:31.581679 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
W20250622 13:51:31.586894 1022 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250622 13:51:31.587051 1022 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250622 13:51:31.592317 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:31.592620 413 tablet_replica.cc:331] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3: stopping tablet replica
I20250622 13:51:31.592937 413 raft_consensus.cc:2241] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.593289 413 raft_consensus.cc:2270] T 9da2656343744cfebe86fefdaf1f0c49 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.594058 413 tablet_replica.cc:331] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3: stopping tablet replica
I20250622 13:51:31.594225 413 raft_consensus.cc:2241] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.594455 413 raft_consensus.cc:2270] T 8f78343995bd4bf58432e3a0af6ed595 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.595167 413 tablet_replica.cc:331] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3: stopping tablet replica
I20250622 13:51:31.595346 413 raft_consensus.cc:2241] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.595546 413 raft_consensus.cc:2270] T 0d87f1c84f02409fb7f2c1e8c0270234 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.596279 413 tablet_replica.cc:331] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3: stopping tablet replica
I20250622 13:51:31.596441 413 raft_consensus.cc:2241] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.596765 413 raft_consensus.cc:2270] T ad5e40f3433e40999825f73f73f4d5fb P 72d517046abc4f898ddb321158a611d3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.597617 413 tablet_replica.cc:331] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3: stopping tablet replica
I20250622 13:51:31.597801 413 raft_consensus.cc:2241] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.597981 413 raft_consensus.cc:2270] T e377e7e836f1484a8bd2f462644c4767 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250622 13:51:31.598387 1117 consensus_peers.cc:487] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 -> Peer e613b3722f774902b7f3f2569a18c925 (127.0.103.65:46271): Couldn't send request to peer e613b3722f774902b7f3f2569a18c925. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:46271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:51:31.598433 413 tablet_replica.cc:331] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3: stopping tablet replica
I20250622 13:51:31.598569 413 raft_consensus.cc:2241] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.598743 413 raft_consensus.cc:2270] T e774fd12bff54d49b10740499f5e86d5 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.599406 413 tablet_replica.cc:331] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3: stopping tablet replica
I20250622 13:51:31.599577 413 raft_consensus.cc:2241] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.599840 413 raft_consensus.cc:2270] T db3c8931bb80491ca282ae5e2d2e5418 P 72d517046abc4f898ddb321158a611d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.600260 413 tablet_replica.cc:331] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3: stopping tablet replica
I20250622 13:51:31.600404 413 raft_consensus.cc:2241] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.600562 413 raft_consensus.cc:2270] T e0764ec090e04b3cb462d33e6e0a95fe P 72d517046abc4f898ddb321158a611d3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.606426 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:51:31.610765 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
W20250622 13:51:31.613751 1022 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250622 13:51:31.614145 1022 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250622 13:51:31.619663 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:31.619890 413 tablet_replica.cc:331] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2: stopping tablet replica
I20250622 13:51:31.620077 413 raft_consensus.cc:2241] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.620268 413 raft_consensus.cc:2270] T 9da2656343744cfebe86fefdaf1f0c49 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.620807 413 tablet_replica.cc:331] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2: stopping tablet replica
I20250622 13:51:31.620973 413 raft_consensus.cc:2241] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.621228 413 raft_consensus.cc:2270] T e0764ec090e04b3cb462d33e6e0a95fe P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.621699 413 tablet_replica.cc:331] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2: stopping tablet replica
I20250622 13:51:31.621840 413 raft_consensus.cc:2241] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.622013 413 raft_consensus.cc:2270] T 0d87f1c84f02409fb7f2c1e8c0270234 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.622571 413 tablet_replica.cc:331] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2: stopping tablet replica
I20250622 13:51:31.622740 413 raft_consensus.cc:2241] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.622900 413 raft_consensus.cc:2270] T e774fd12bff54d49b10740499f5e86d5 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.623450 413 tablet_replica.cc:331] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2: stopping tablet replica
I20250622 13:51:31.623601 413 raft_consensus.cc:2241] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.623746 413 raft_consensus.cc:2270] T e377e7e836f1484a8bd2f462644c4767 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.624195 413 tablet_replica.cc:331] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2: stopping tablet replica
I20250622 13:51:31.624347 413 raft_consensus.cc:2241] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.624505 413 raft_consensus.cc:2270] T ad5e40f3433e40999825f73f73f4d5fb P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.625161 413 tablet_replica.cc:331] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2: stopping tablet replica
I20250622 13:51:31.625295 413 raft_consensus.cc:2241] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.625624 413 raft_consensus.cc:2270] T 8f78343995bd4bf58432e3a0af6ed595 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.626217 413 tablet_replica.cc:331] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2: stopping tablet replica
I20250622 13:51:31.626363 413 raft_consensus.cc:2241] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:31.626497 413 raft_consensus.cc:2270] T db3c8931bb80491ca282ae5e2d2e5418 P 3b582b3d93dc4a7ebc3128b41d1895f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.641547 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:51:31.645255 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250622 13:51:31.653416 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:31.666213 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250622 13:51:31.668736 413 master.cc:561] Master@127.0.103.126:35365 shutting down...
I20250622 13:51:31.676765 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:31.676981 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:31.677076 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P d6a8e62d743a4526861a52bdd8454d6d: stopping tablet replica
I20250622 13:51:31.680825 413 master.cc:583] Master@127.0.103.126:35365 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7371 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20250622 13:51:31.697026 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:35915,127.0.103.125:42015,127.0.103.124:38517
I20250622 13:51:31.697695 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:31.700038 1447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:31.700124 1448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:31.700740 1450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.701574 413 server_base.cc:1048] running on GCE node
I20250622 13:51:31.701856 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:31.701925 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:31.702003 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600291701968 us; error 0 us; skew 500 ppm
I20250622 13:51:31.702170 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:31.703477 413 webserver.cc:469] Webserver started at http://127.0.103.126:43815/ using document root <none> and password file <none>
I20250622 13:51:31.703696 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:31.703792 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:31.703936 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:31.704574 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "6a0038f47cbb4ac7860a94be781178e0"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.707161 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250622 13:51:31.708802 1455 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:31.709158 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:51:31.709273 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "6a0038f47cbb4ac7860a94be781178e0"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.709434 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:31.722011 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:31.722720 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:31.736495 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:35915
I20250622 13:51:31.736546 1506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:35915 every 8 connection(s)
I20250622 13:51:31.737840 1507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:31.737898 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:31.740546 1509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:31.740545 1510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.741073 1507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } has no permanent_uuid. Determining permanent_uuid...
W20250622 13:51:31.741469 1512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.741782 413 server_base.cc:1048] running on GCE node
I20250622 13:51:31.742134 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:31.742228 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:31.742313 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600291742271 us; error 0 us; skew 500 ppm
I20250622 13:51:31.742528 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:31.743916 413 webserver.cc:469] Webserver started at http://127.0.103.125:46433/ using document root <none> and password file <none>
I20250622 13:51:31.744138 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:31.744238 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:31.744385 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:31.745013 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root/instance:
uuid: "b96e578b1a0d4aee95db964ac6b66b8b"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.747725 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:31.748229 1507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:31.749606 1519 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:31.750062 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:51:31.750177 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root
uuid: "b96e578b1a0d4aee95db964ac6b66b8b"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.750303 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250622 13:51:31.754803 1507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.125:42015: Network error: Client connection negotiation failed: client connection to 127.0.103.125:42015: connect: Connection refused (error 111)
I20250622 13:51:31.766961 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:31.767562 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:31.779506 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.125:42015
I20250622 13:51:31.779548 1571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.125:42015 every 8 connection(s)
I20250622 13:51:31.780831 1572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:31.780927 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:31.783433 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:31.783794 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.783906 1572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:31.785773 413 server_base.cc:1048] running on GCE node
W20250622 13:51:31.786554 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.787015 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:31.787091 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:31.787158 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600291787116 us; error 0 us; skew 500 ppm
I20250622 13:51:31.787329 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:31.788919 413 webserver.cc:469] Webserver started at http://127.0.103.124:46505/ using document root <none> and password file <none>
I20250622 13:51:31.789199 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:31.789317 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:31.789486 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:31.790287 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root/instance:
uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.791905 1572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:31.794589 413 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20250622 13:51:31.796350 1584 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:31.796706 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:51:31.796798 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root
uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.796968 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:31.799639 1572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
W20250622 13:51:31.801584 1572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.124:38517: Network error: Client connection negotiation failed: client connection to 127.0.103.124:38517: connect: Connection refused (error 111)
I20250622 13:51:31.820039 1507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } attempt: 1
I20250622 13:51:31.825783 1507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
W20250622 13:51:31.827589 1507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.124:38517: Network error: Client connection negotiation failed: client connection to 127.0.103.124:38517: connect: Connection refused (error 111)
I20250622 13:51:31.827730 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:31.828316 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:31.842250 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.124:38517
I20250622 13:51:31.842284 1636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.124:38517 every 8 connection(s)
I20250622 13:51:31.843266 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:51:31.843742 1637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:31.845984 1637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:31.849758 1572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } attempt: 1
I20250622 13:51:31.852195 1637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:31.858721 1637 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
I20250622 13:51:31.860430 1572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: Bootstrap starting.
I20250622 13:51:31.862052 1572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:31.864743 1572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: No bootstrap required, opened a new log
I20250622 13:51:31.865610 1572 raft_consensus.cc:357] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:31.865825 1572 raft_consensus.cc:383] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:31.865913 1572 raft_consensus.cc:738] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96e578b1a0d4aee95db964ac6b66b8b, State: Initialized, Role: FOLLOWER
I20250622 13:51:31.866160 1572 consensus_queue.cc:260] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:31.866792 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:31.866988 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:31.867089 1572 sys_catalog.cc:564] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:51:31.867681 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739: Bootstrap starting.
I20250622 13:51:31.869813 1637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:31.871901 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739: No bootstrap required, opened a new log
I20250622 13:51:31.872956 1637 raft_consensus.cc:357] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
W20250622 13:51:31.873039 1654 catalog_manager.cc:1560] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250622 13:51:31.873111 1637 raft_consensus.cc:383] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250622 13:51:31.873148 1654 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250622 13:51:31.873208 1637 raft_consensus.cc:738] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf2c2f3f8964fc4b6b08aebc87fe739, State: Initialized, Role: FOLLOWER
I20250622 13:51:31.873476 1637 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:31.874007 1655 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:31.874150 1655 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:31.874176 1637 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: configured and running, proceeding with master startup.
W20250622 13:51:31.877981 1666 catalog_manager.cc:1560] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250622 13:51:31.878065 1666 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250622 13:51:31.886790 1507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } attempt: 1
I20250622 13:51:31.894860 1507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0: Bootstrap starting.
I20250622 13:51:31.896179 1507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:31.897894 1507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0: No bootstrap required, opened a new log
I20250622 13:51:31.898519 1507 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:31.898670 1507 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:31.898726 1507 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0038f47cbb4ac7860a94be781178e0, State: Initialized, Role: FOLLOWER
I20250622 13:51:31.898948 1507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:31.899490 1668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:31.899658 1668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:31.899672 1507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:51:31.903863 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250622 13:51:31.904024 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250622 13:51:31.904891 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:31.905217 1679 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250622 13:51:31.905328 1679 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250622 13:51:31.907830 1680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:31.908284 1681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:31.908800 1683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.909150 413 server_base.cc:1048] running on GCE node
I20250622 13:51:31.909446 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:31.909538 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:31.909600 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600291909581 us; error 0 us; skew 500 ppm
I20250622 13:51:31.909771 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:31.911180 413 webserver.cc:469] Webserver started at http://127.0.103.65:39821/ using document root <none> and password file <none>
I20250622 13:51:31.911429 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:31.911520 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:31.911674 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:31.912323 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "1c960a62d33a43ef9bc3bbc055974db6"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.915469 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250622 13:51:31.917104 1688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:31.917500 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:51:31.917593 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "1c960a62d33a43ef9bc3bbc055974db6"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.917697 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:31.932947 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:31.933552 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:31.934126 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:31.934933 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:31.935005 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:31.935082 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:31.935137 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:31.951426 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38479
I20250622 13:51:31.951465 1750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38479 every 8 connection(s)
I20250622 13:51:31.954898 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:31.962486 1758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.962815 1751 heartbeater.cc:344] Connected to a master server at 127.0.103.124:38517
I20250622 13:51:31.962963 1751 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:31.963410 1751 heartbeater.cc:507] Master 127.0.103.124:38517 requested a full tablet report, sending...
I20250622 13:51:31.964071 1753 heartbeater.cc:344] Connected to a master server at 127.0.103.125:42015
I20250622 13:51:31.964201 1753 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:31.964644 1602 ts_manager.cc:194] Registered new tserver with Master: 1c960a62d33a43ef9bc3bbc055974db6 (127.0.103.65:38479)
I20250622 13:51:31.964687 1753 heartbeater.cc:507] Master 127.0.103.125:42015 requested a full tablet report, sending...
I20250622 13:51:31.966421 1537 ts_manager.cc:194] Registered new tserver with Master: 1c960a62d33a43ef9bc3bbc055974db6 (127.0.103.65:38479)
W20250622 13:51:31.966881 1759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.967559 1752 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35915
I20250622 13:51:31.967741 1752 heartbeater.cc:461] Registering TS with master...
W20250622 13:51:31.968348 1761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:31.968385 1752 heartbeater.cc:507] Master 127.0.103.126:35915 requested a full tablet report, sending...
I20250622 13:51:31.968791 413 server_base.cc:1048] running on GCE node
I20250622 13:51:31.969108 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:31.969202 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:31.969287 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600291969248 us; error 0 us; skew 500 ppm
I20250622 13:51:31.969497 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:31.969496 1472 ts_manager.cc:194] Registered new tserver with Master: 1c960a62d33a43ef9bc3bbc055974db6 (127.0.103.65:38479)
I20250622 13:51:31.970836 413 webserver.cc:469] Webserver started at http://127.0.103.66:43647/ using document root <none> and password file <none>
I20250622 13:51:31.971043 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:31.971143 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:31.971276 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:31.971902 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "3ea2b16ce8ad402e89d6f9de6b9b24ec"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.974334 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:31.975994 1766 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:31.976433 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:51:31.976545 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "3ea2b16ce8ad402e89d6f9de6b9b24ec"
format_stamp: "Formatted at 2025-06-22 13:51:31 on dist-test-slave-tmv0"
I20250622 13:51:31.976668 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:31.993801 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:31.994382 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:31.995402 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:31.996272 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:31.996342 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:31.996454 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:31.996508 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:32.012398 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:40677
I20250622 13:51:32.012454 1828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:40677 every 8 connection(s)
I20250622 13:51:32.016736 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250622 13:51:32.026014 1830 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35915
I20250622 13:51:32.026038 1829 heartbeater.cc:344] Connected to a master server at 127.0.103.124:38517
I20250622 13:51:32.026281 1829 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:32.026286 1830 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:32.026983 1830 heartbeater.cc:507] Master 127.0.103.126:35915 requested a full tablet report, sending...
I20250622 13:51:32.027004 1829 heartbeater.cc:507] Master 127.0.103.124:38517 requested a full tablet report, sending...
I20250622 13:51:32.028388 1472 ts_manager.cc:194] Registered new tserver with Master: 3ea2b16ce8ad402e89d6f9de6b9b24ec (127.0.103.66:40677)
I20250622 13:51:32.028384 1602 ts_manager.cc:194] Registered new tserver with Master: 3ea2b16ce8ad402e89d6f9de6b9b24ec (127.0.103.66:40677)
I20250622 13:51:32.030004 1831 heartbeater.cc:344] Connected to a master server at 127.0.103.125:42015
I20250622 13:51:32.030200 1831 heartbeater.cc:461] Registering TS with master...
W20250622 13:51:32.030684 1836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:32.030982 1831 heartbeater.cc:507] Master 127.0.103.125:42015 requested a full tablet report, sending...
W20250622 13:51:32.031692 1837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:32.032758 413 server_base.cc:1048] running on GCE node
I20250622 13:51:32.032816 1537 ts_manager.cc:194] Registered new tserver with Master: 3ea2b16ce8ad402e89d6f9de6b9b24ec (127.0.103.66:40677)
W20250622 13:51:32.033078 1839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:32.033468 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:32.033558 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:32.033628 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600292033610 us; error 0 us; skew 500 ppm
I20250622 13:51:32.033789 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:32.035177 413 webserver.cc:469] Webserver started at http://127.0.103.67:33699/ using document root <none> and password file <none>
I20250622 13:51:32.035900 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:32.035995 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:32.036144 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:32.036754 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "113166b872684374bebe24cc2b1921f9"
format_stamp: "Formatted at 2025-06-22 13:51:32 on dist-test-slave-tmv0"
I20250622 13:51:32.039192 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250622 13:51:32.040707 1844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:32.041051 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:51:32.041169 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "113166b872684374bebe24cc2b1921f9"
format_stamp: "Formatted at 2025-06-22 13:51:32 on dist-test-slave-tmv0"
I20250622 13:51:32.041288 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:32.063042 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:32.063647 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:32.064209 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:32.065081 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:32.065181 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:32.065279 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:32.065371 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:32.082929 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:37677
I20250622 13:51:32.082978 1906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:37677 every 8 connection(s)
I20250622 13:51:32.092098 1907 heartbeater.cc:344] Connected to a master server at 127.0.103.124:38517
I20250622 13:51:32.092296 1907 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:32.092877 1907 heartbeater.cc:507] Master 127.0.103.124:38517 requested a full tablet report, sending...
I20250622 13:51:32.094040 1602 ts_manager.cc:194] Registered new tserver with Master: 113166b872684374bebe24cc2b1921f9 (127.0.103.67:37677)
I20250622 13:51:32.098786 1909 heartbeater.cc:344] Connected to a master server at 127.0.103.125:42015
I20250622 13:51:32.098855 1908 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35915
I20250622 13:51:32.098966 1908 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:32.098970 1909 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:32.099566 1909 heartbeater.cc:507] Master 127.0.103.125:42015 requested a full tablet report, sending...
I20250622 13:51:32.099588 1908 heartbeater.cc:507] Master 127.0.103.126:35915 requested a full tablet report, sending...
I20250622 13:51:32.100584 1472 ts_manager.cc:194] Registered new tserver with Master: 113166b872684374bebe24cc2b1921f9 (127.0.103.67:37677)
I20250622 13:51:32.100701 1537 ts_manager.cc:194] Registered new tserver with Master: 113166b872684374bebe24cc2b1921f9 (127.0.103.67:37677)
I20250622 13:51:32.101714 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016013205s
I20250622 13:51:32.151305 1655 raft_consensus.cc:491] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:32.151516 1655 raft_consensus.cc:513] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:32.152159 1655 leader_election.cc:290] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0038f47cbb4ac7860a94be781178e0 (127.0.103.126:35915), b96e578b1a0d4aee95db964ac6b66b8b (127.0.103.125:42015)
I20250622 13:51:32.152719 1482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0038f47cbb4ac7860a94be781178e0" is_pre_election: true
I20250622 13:51:32.152740 1547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" is_pre_election: true
I20250622 13:51:32.153039 1547 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddf2c2f3f8964fc4b6b08aebc87fe739 in term 0.
I20250622 13:51:32.153036 1482 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddf2c2f3f8964fc4b6b08aebc87fe739 in term 0.
I20250622 13:51:32.153519 1585 leader_election.cc:304] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b96e578b1a0d4aee95db964ac6b66b8b, ddf2c2f3f8964fc4b6b08aebc87fe739; no voters:
I20250622 13:51:32.153750 1655 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:32.153857 1655 raft_consensus.cc:491] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:32.153915 1655 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:32.155774 1655 raft_consensus.cc:513] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:32.156270 1655 leader_election.cc:290] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0038f47cbb4ac7860a94be781178e0 (127.0.103.126:35915), b96e578b1a0d4aee95db964ac6b66b8b (127.0.103.125:42015)
I20250622 13:51:32.156721 1547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96e578b1a0d4aee95db964ac6b66b8b"
I20250622 13:51:32.156700 1482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0038f47cbb4ac7860a94be781178e0"
I20250622 13:51:32.156930 1547 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:32.156929 1482 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:32.159824 1547 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddf2c2f3f8964fc4b6b08aebc87fe739 in term 1.
I20250622 13:51:32.159879 1482 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddf2c2f3f8964fc4b6b08aebc87fe739 in term 1.
I20250622 13:51:32.160205 1585 leader_election.cc:304] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b96e578b1a0d4aee95db964ac6b66b8b, ddf2c2f3f8964fc4b6b08aebc87fe739; no voters:
I20250622 13:51:32.160461 1655 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:32.160905 1655 raft_consensus.cc:695] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 1 LEADER]: Becoming Leader. State: Replica: ddf2c2f3f8964fc4b6b08aebc87fe739, State: Running, Role: LEADER
I20250622 13:51:32.161159 1655 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:32.162818 1915 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf2c2f3f8964fc4b6b08aebc87fe739. Latest consensus state: current_term: 1 leader_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:32.163087 1915 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:32.163681 1917 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:51:32.164994 1917 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:51:32.167681 1482 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 1 FOLLOWER]: Refusing update from remote peer ddf2c2f3f8964fc4b6b08aebc87fe739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:32.167698 1547 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 1 FOLLOWER]: Refusing update from remote peer ddf2c2f3f8964fc4b6b08aebc87fe739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:32.168198 1655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:32.168380 1915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:32.174039 1668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf2c2f3f8964fc4b6b08aebc87fe739. Latest consensus state: current_term: 1 leader_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:32.174333 1668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:32.175679 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf2c2f3f8964fc4b6b08aebc87fe739. Latest consensus state: current_term: 1 leader_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:32.175925 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:32.176656 1915 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:32.176862 1915 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:32.177588 1917 catalog_manager.cc:1349] Generated new cluster ID: f311730398534b33abf1d2ad79b42f35
I20250622 13:51:32.177816 1917 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:51:32.178577 1915 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:32.178843 1915 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:32.179535 1668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:32.179715 1668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:32.182000 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:32.182214 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:32.201426 1917 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:51:32.202550 1917 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:51:32.217685 1917 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739: Generated new TSK 0
I20250622 13:51:32.218036 1917 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:51:32.874701 1654 catalog_manager.cc:1261] Loaded cluster ID: f311730398534b33abf1d2ad79b42f35
I20250622 13:51:32.874818 1654 catalog_manager.cc:1554] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: loading cluster ID for follower catalog manager: success
I20250622 13:51:32.877251 1654 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: acquiring CA information for follower catalog manager: success
I20250622 13:51:32.878819 1654 catalog_manager.cc:1604] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250622 13:51:32.906566 1679 catalog_manager.cc:1261] Loaded cluster ID: f311730398534b33abf1d2ad79b42f35
I20250622 13:51:32.906673 1679 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0: loading cluster ID for follower catalog manager: success
I20250622 13:51:32.908465 1679 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0: acquiring CA information for follower catalog manager: success
I20250622 13:51:32.909479 1679 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250622 13:51:32.967476 1602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58008
I20250622 13:51:33.031127 1602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58018
I20250622 13:51:33.096505 1602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58020
I20250622 13:51:33.877216 1665 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:33.968964 1751 heartbeater.cc:499] Master 127.0.103.124:38517 was elected leader, sending a full tablet report...
I20250622 13:51:34.032248 1829 heartbeater.cc:499] Master 127.0.103.124:38517 was elected leader, sending a full tablet report...
I20250622 13:51:34.097893 1907 heartbeater.cc:499] Master 127.0.103.124:38517 was elected leader, sending a full tablet report...
I20250622 13:51:34.362154 413 test_util.cc:276] Using random seed: 1689283606
I20250622 13:51:34.379130 1602 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58022:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 13:51:34.428659 1716 tablet_service.cc:1468] Processing CreateTablet for tablet c22b510dc945468eac619949b6f40cfc (DEFAULT_TABLE table=test-workload [id=67c84944a9154ada9d7d514942bd05fd]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250622 13:51:34.429237 1716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22b510dc945468eac619949b6f40cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:34.430866 1794 tablet_service.cc:1468] Processing CreateTablet for tablet 0301213c2eed40bf95cb1d5c84c44867 (DEFAULT_TABLE table=test-workload [id=67c84944a9154ada9d7d514942bd05fd]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250622 13:51:34.431448 1794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0301213c2eed40bf95cb1d5c84c44867. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:34.433195 1872 tablet_service.cc:1468] Processing CreateTablet for tablet 0e150c10c519494393387aaffc99b623 (DEFAULT_TABLE table=test-workload [id=67c84944a9154ada9d7d514942bd05fd]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250622 13:51:34.433718 1872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e150c10c519494393387aaffc99b623. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:34.437104 1954 tablet_bootstrap.cc:492] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6: Bootstrap starting.
I20250622 13:51:34.437661 1955 tablet_bootstrap.cc:492] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec: Bootstrap starting.
I20250622 13:51:34.439476 1954 tablet_bootstrap.cc:654] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:34.440227 1955 tablet_bootstrap.cc:654] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:34.443053 1955 tablet_bootstrap.cc:492] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec: No bootstrap required, opened a new log
I20250622 13:51:34.443200 1955 ts_tablet_manager.cc:1397] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20250622 13:51:34.443429 1954 tablet_bootstrap.cc:492] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6: No bootstrap required, opened a new log
I20250622 13:51:34.443580 1954 ts_tablet_manager.cc:1397] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250622 13:51:34.443957 1955 raft_consensus.cc:357] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea2b16ce8ad402e89d6f9de6b9b24ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40677 } }
I20250622 13:51:34.444089 1955 raft_consensus.cc:383] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:34.444196 1955 raft_consensus.cc:738] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea2b16ce8ad402e89d6f9de6b9b24ec, State: Initialized, Role: FOLLOWER
I20250622 13:51:34.444398 1957 tablet_bootstrap.cc:492] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9: Bootstrap starting.
I20250622 13:51:34.444422 1955 consensus_queue.cc:260] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea2b16ce8ad402e89d6f9de6b9b24ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40677 } }
I20250622 13:51:34.444633 1955 raft_consensus.cc:397] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:34.444725 1955 raft_consensus.cc:491] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:34.444832 1955 raft_consensus.cc:3058] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:34.445446 1954 raft_consensus.cc:357] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c960a62d33a43ef9bc3bbc055974db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38479 } }
I20250622 13:51:34.445639 1954 raft_consensus.cc:383] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:34.445735 1954 raft_consensus.cc:738] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c960a62d33a43ef9bc3bbc055974db6, State: Initialized, Role: FOLLOWER
I20250622 13:51:34.445973 1954 consensus_queue.cc:260] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c960a62d33a43ef9bc3bbc055974db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38479 } }
I20250622 13:51:34.446175 1954 raft_consensus.cc:397] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:34.446256 1957 tablet_bootstrap.cc:654] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:34.446295 1954 raft_consensus.cc:491] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:34.446902 1954 raft_consensus.cc:3058] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:34.447883 1955 raft_consensus.cc:513] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea2b16ce8ad402e89d6f9de6b9b24ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40677 } }
I20250622 13:51:34.448155 1955 leader_election.cc:304] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ea2b16ce8ad402e89d6f9de6b9b24ec; no voters:
I20250622 13:51:34.448791 1955 leader_election.cc:290] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:34.448884 1957 tablet_bootstrap.cc:492] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9: No bootstrap required, opened a new log
I20250622 13:51:34.448947 1960 raft_consensus.cc:2802] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:34.449075 1957 ts_tablet_manager.cc:1397] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250622 13:51:34.449837 1954 raft_consensus.cc:513] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c960a62d33a43ef9bc3bbc055974db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38479 } }
I20250622 13:51:34.449923 1957 raft_consensus.cc:357] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113166b872684374bebe24cc2b1921f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37677 } }
I20250622 13:51:34.450131 1957 raft_consensus.cc:383] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:34.450290 1957 raft_consensus.cc:738] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113166b872684374bebe24cc2b1921f9, State: Initialized, Role: FOLLOWER
I20250622 13:51:34.450109 1954 leader_election.cc:304] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c960a62d33a43ef9bc3bbc055974db6; no voters:
I20250622 13:51:34.450790 1957 consensus_queue.cc:260] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113166b872684374bebe24cc2b1921f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37677 } }
I20250622 13:51:34.451087 1963 raft_consensus.cc:2802] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:34.451157 1954 leader_election.cc:290] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:34.451047 1957 raft_consensus.cc:397] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:34.451649 1957 raft_consensus.cc:491] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:34.451898 1957 raft_consensus.cc:3058] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:34.455708 1957 raft_consensus.cc:513] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113166b872684374bebe24cc2b1921f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37677 } }
I20250622 13:51:34.455960 1955 ts_tablet_manager.cc:1428] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
I20250622 13:51:34.455999 1963 raft_consensus.cc:695] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c960a62d33a43ef9bc3bbc055974db6, State: Running, Role: LEADER
I20250622 13:51:34.456784 1963 consensus_queue.cc:237] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c960a62d33a43ef9bc3bbc055974db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38479 } }
I20250622 13:51:34.457437 1954 ts_tablet_manager.cc:1428] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20250622 13:51:34.457659 1960 raft_consensus.cc:695] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 1 LEADER]: Becoming Leader. State: Replica: 3ea2b16ce8ad402e89d6f9de6b9b24ec, State: Running, Role: LEADER
I20250622 13:51:34.456252 1957 leader_election.cc:304] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 113166b872684374bebe24cc2b1921f9; no voters:
I20250622 13:51:34.457943 1960 consensus_queue.cc:237] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea2b16ce8ad402e89d6f9de6b9b24ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40677 } }
I20250622 13:51:34.459043 1957 leader_election.cc:290] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:34.459246 1967 raft_consensus.cc:2802] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:34.461889 1602 catalog_manager.cc:5582] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ea2b16ce8ad402e89d6f9de6b9b24ec (127.0.103.66). New cstate: current_term: 1 leader_uuid: "3ea2b16ce8ad402e89d6f9de6b9b24ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea2b16ce8ad402e89d6f9de6b9b24ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40677 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:34.466548 1967 raft_consensus.cc:695] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 1 LEADER]: Becoming Leader. State: Replica: 113166b872684374bebe24cc2b1921f9, State: Running, Role: LEADER
I20250622 13:51:34.466874 1967 consensus_queue.cc:237] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113166b872684374bebe24cc2b1921f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37677 } }
I20250622 13:51:34.469628 1601 catalog_manager.cc:5582] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 113166b872684374bebe24cc2b1921f9 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "113166b872684374bebe24cc2b1921f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113166b872684374bebe24cc2b1921f9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37677 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:34.476846 1957 ts_tablet_manager.cc:1428] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9: Time spent starting tablet: real 0.028s user 0.005s sys 0.006s
I20250622 13:51:34.484589 1602 catalog_manager.cc:5582] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c960a62d33a43ef9bc3bbc055974db6 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "1c960a62d33a43ef9bc3bbc055974db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c960a62d33a43ef9bc3bbc055974db6" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38479 } health_report { overall_health: HEALTHY } } }
I20250622 13:51:34.503940 413 master.cc:561] Master@127.0.103.124:38517 shutting down...
I20250622 13:51:34.510970 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:34.511360 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:34.511499 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P ddf2c2f3f8964fc4b6b08aebc87fe739: stopping tablet replica
I20250622 13:51:34.526897 413 master.cc:583] Master@127.0.103.124:38517 shutdown complete.
W20250622 13:51:35.473887 1829 heartbeater.cc:646] Failed to heartbeat to 127.0.103.124:38517 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.103.124:38517: connect: Connection refused (error 111)
I20250622 13:51:35.998524 1980 raft_consensus.cc:491] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ddf2c2f3f8964fc4b6b08aebc87fe739)
I20250622 13:51:35.998726 1980 raft_consensus.cc:513] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:35.999377 1980 leader_election.cc:290] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a0038f47cbb4ac7860a94be781178e0 (127.0.103.126:35915), ddf2c2f3f8964fc4b6b08aebc87fe739 (127.0.103.124:38517)
I20250622 13:51:35.999953 1482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "6a0038f47cbb4ac7860a94be781178e0" is_pre_election: true
I20250622 13:51:36.000195 1482 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b96e578b1a0d4aee95db964ac6b66b8b in term 1.
I20250622 13:51:36.000873 1521 leader_election.cc:304] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0038f47cbb4ac7860a94be781178e0, b96e578b1a0d4aee95db964ac6b66b8b; no voters:
I20250622 13:51:36.001220 1980 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:51:36.001379 1980 raft_consensus.cc:491] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 1 FOLLOWER]: Starting leader election (detected failure of leader ddf2c2f3f8964fc4b6b08aebc87fe739)
I20250622 13:51:36.001541 1980 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:51:36.005254 1980 raft_consensus.cc:513] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:36.006261 1980 leader_election.cc:290] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [CANDIDATE]: Term 2 election: Requested vote from peers 6a0038f47cbb4ac7860a94be781178e0 (127.0.103.126:35915), ddf2c2f3f8964fc4b6b08aebc87fe739 (127.0.103.124:38517)
I20250622 13:51:36.006533 1982 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ddf2c2f3f8964fc4b6b08aebc87fe739)
I20250622 13:51:36.006749 1982 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:36.006781 1482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "6a0038f47cbb4ac7860a94be781178e0"
I20250622 13:51:36.006987 1482 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:51:36.007341 1982 leader_election.cc:290] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b96e578b1a0d4aee95db964ac6b66b8b (127.0.103.125:42015), ddf2c2f3f8964fc4b6b08aebc87fe739 (127.0.103.124:38517)
I20250622 13:51:36.007711 1547 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a0038f47cbb4ac7860a94be781178e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" is_pre_election: true
I20250622 13:51:36.007961 1547 raft_consensus.cc:2391] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a0038f47cbb4ac7860a94be781178e0 in current term 2: Already voted for candidate b96e578b1a0d4aee95db964ac6b66b8b in this term.
W20250622 13:51:36.008322 1524 leader_election.cc:336] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ddf2c2f3f8964fc4b6b08aebc87fe739 (127.0.103.124:38517): Network error: Client connection negotiation failed: client connection to 127.0.103.124:38517: connect: Connection refused (error 111)
W20250622 13:51:36.009630 1524 leader_election.cc:336] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ddf2c2f3f8964fc4b6b08aebc87fe739 (127.0.103.124:38517): Network error: Client connection negotiation failed: client connection to 127.0.103.124:38517: connect: Connection refused (error 111)
I20250622 13:51:36.010180 1482 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b96e578b1a0d4aee95db964ac6b66b8b in term 2.
W20250622 13:51:36.010648 1459 leader_election.cc:336] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ddf2c2f3f8964fc4b6b08aebc87fe739 (127.0.103.124:38517): Network error: Client connection negotiation failed: client connection to 127.0.103.124:38517: connect: Connection refused (error 111)
I20250622 13:51:36.010656 1521 leader_election.cc:304] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0038f47cbb4ac7860a94be781178e0, b96e578b1a0d4aee95db964ac6b66b8b; no voters: ddf2c2f3f8964fc4b6b08aebc87fe739
I20250622 13:51:36.010773 1459 leader_election.cc:304] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a0038f47cbb4ac7860a94be781178e0; no voters: b96e578b1a0d4aee95db964ac6b66b8b, ddf2c2f3f8964fc4b6b08aebc87fe739
I20250622 13:51:36.010999 1980 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:51:36.011037 1982 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250622 13:51:36.011349 1980 raft_consensus.cc:695] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 2 LEADER]: Becoming Leader. State: Replica: b96e578b1a0d4aee95db964ac6b66b8b, State: Running, Role: LEADER
I20250622 13:51:36.011605 1980 consensus_queue.cc:237] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } }
I20250622 13:51:36.012706 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b96e578b1a0d4aee95db964ac6b66b8b. Latest consensus state: current_term: 2 leader_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:36.012926 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:36.013438 1988 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:51:36.014997 1988 catalog_manager.cc:671] Loaded metadata for table test-workload [id=67c84944a9154ada9d7d514942bd05fd]
I20250622 13:51:36.016310 1988 tablet_loader.cc:96] loaded metadata for tablet 0301213c2eed40bf95cb1d5c84c44867 (table test-workload [id=67c84944a9154ada9d7d514942bd05fd])
I20250622 13:51:36.016634 1988 tablet_loader.cc:96] loaded metadata for tablet 0e150c10c519494393387aaffc99b623 (table test-workload [id=67c84944a9154ada9d7d514942bd05fd])
I20250622 13:51:36.016930 1988 tablet_loader.cc:96] loaded metadata for tablet c22b510dc945468eac619949b6f40cfc (table test-workload [id=67c84944a9154ada9d7d514942bd05fd])
I20250622 13:51:36.017144 1988 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:51:36.017961 1988 catalog_manager.cc:1261] Loaded cluster ID: f311730398534b33abf1d2ad79b42f35
I20250622 13:51:36.018030 1988 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:51:36.019155 1988 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:51:36.019963 1988 catalog_manager.cc:5966] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: Loaded TSK: 0
I20250622 13:51:36.020553 1988 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:51:36.494836 1482 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 2 FOLLOWER]: Refusing update from remote peer b96e578b1a0d4aee95db964ac6b66b8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250622 13:51:36.495412 1986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250622 13:51:36.496793 1982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b96e578b1a0d4aee95db964ac6b66b8b. Latest consensus state: current_term: 2 leader_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:36.496992 1982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: This master's current role is: FOLLOWER
I20250622 13:51:36.498800 1980 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:36.499212 1980 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [sys.catalog]: This master's current role is: LEADER
W20250622 13:51:36.500149 1524 consensus_peers.cc:487] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b -> Peer ddf2c2f3f8964fc4b6b08aebc87fe739 (127.0.103.124:38517): Couldn't send request to peer ddf2c2f3f8964fc4b6b08aebc87fe739. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.124:38517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:51:36.501025 1982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0038f47cbb4ac7860a94be781178e0" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35915 } } peers { permanent_uuid: "b96e578b1a0d4aee95db964ac6b66b8b" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 42015 } } peers { permanent_uuid: "ddf2c2f3f8964fc4b6b08aebc87fe739" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 38517 } } }
I20250622 13:51:36.501166 1982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [sys.catalog]: This master's current role is: FOLLOWER
W20250622 13:51:37.017701 1524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.124:38517: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20250622 13:51:37.469434 1831 heartbeater.cc:499] Master 127.0.103.125:42015 was elected leader, sending a full tablet report...
I20250622 13:51:37.485093 1909 heartbeater.cc:499] Master 127.0.103.125:42015 was elected leader, sending a full tablet report...
I20250622 13:51:37.494540 1753 heartbeater.cc:499] Master 127.0.103.125:42015 was elected leader, sending a full tablet report...
I20250622 13:51:37.872277 1653 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:37.872416 1653 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:38.067605 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:51:38.075865 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:38.076162 413 tablet_replica.cc:331] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6: stopping tablet replica
I20250622 13:51:38.076347 413 raft_consensus.cc:2241] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:38.076510 413 raft_consensus.cc:2270] T c22b510dc945468eac619949b6f40cfc P 1c960a62d33a43ef9bc3bbc055974db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:38.089915 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:51:38.092676 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:51:38.100975 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:38.101269 413 tablet_replica.cc:331] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec: stopping tablet replica
I20250622 13:51:38.101486 413 raft_consensus.cc:2241] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:38.101714 413 raft_consensus.cc:2270] T 0301213c2eed40bf95cb1d5c84c44867 P 3ea2b16ce8ad402e89d6f9de6b9b24ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:38.115249 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:51:38.117918 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:51:38.126442 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:51:38.126735 413 tablet_replica.cc:331] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9: stopping tablet replica
I20250622 13:51:38.126892 413 raft_consensus.cc:2241] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:51:38.127050 413 raft_consensus.cc:2270] T 0e150c10c519494393387aaffc99b623 P 113166b872684374bebe24cc2b1921f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:38.140331 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:51:38.142931 413 master.cc:561] Master@127.0.103.126:35915 shutting down...
I20250622 13:51:38.150573 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:38.150810 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:38.150951 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6a0038f47cbb4ac7860a94be781178e0: stopping tablet replica
I20250622 13:51:38.164618 413 master.cc:583] Master@127.0.103.126:35915 shutdown complete.
I20250622 13:51:38.168130 413 master.cc:561] Master@127.0.103.125:42015 shutting down...
I20250622 13:51:38.175452 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:51:38.175853 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:38.175972 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P b96e578b1a0d4aee95db964ac6b66b8b: stopping tablet replica
I20250622 13:51:38.191430 413 master.cc:583] Master@127.0.103.125:42015 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6506 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20250622 13:51:38.203873 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:41021
I20250622 13:51:38.204408 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:38.207087 2003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:38.207273 2004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:38.207386 413 server_base.cc:1048] running on GCE node
W20250622 13:51:38.207314 2006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:38.207962 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:38.208036 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:38.208103 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600298208085 us; error 0 us; skew 500 ppm
I20250622 13:51:38.208261 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:38.212713 413 webserver.cc:469] Webserver started at http://127.0.103.126:34935/ using document root <none> and password file <none>
I20250622 13:51:38.212960 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:38.213065 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:38.213199 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:38.213874 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "bd9be859342c4cd98a85f651fc9789a3"
format_stamp: "Formatted at 2025-06-22 13:51:38 on dist-test-slave-tmv0"
I20250622 13:51:38.216305 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:38.217895 2011 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.218312 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:51:38.218402 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "bd9be859342c4cd98a85f651fc9789a3"
format_stamp: "Formatted at 2025-06-22 13:51:38 on dist-test-slave-tmv0"
I20250622 13:51:38.218521 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:38.242079 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:38.242683 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:38.256446 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:41021
I20250622 13:51:38.256520 2062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:41021 every 8 connection(s)
I20250622 13:51:38.258244 2063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:38.262519 2063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3: Bootstrap starting.
I20250622 13:51:38.264022 2063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:38.265787 2063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3: No bootstrap required, opened a new log
I20250622 13:51:38.266402 2063 raft_consensus.cc:357] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9be859342c4cd98a85f651fc9789a3" member_type: VOTER }
I20250622 13:51:38.266523 2063 raft_consensus.cc:383] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:38.266587 2063 raft_consensus.cc:738] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd9be859342c4cd98a85f651fc9789a3, State: Initialized, Role: FOLLOWER
I20250622 13:51:38.266785 2063 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9be859342c4cd98a85f651fc9789a3" member_type: VOTER }
I20250622 13:51:38.266918 2063 raft_consensus.cc:397] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:51:38.266973 2063 raft_consensus.cc:491] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:51:38.267045 2063 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:38.268594 2063 raft_consensus.cc:513] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9be859342c4cd98a85f651fc9789a3" member_type: VOTER }
I20250622 13:51:38.268796 2063 leader_election.cc:304] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd9be859342c4cd98a85f651fc9789a3; no voters:
I20250622 13:51:38.269167 2063 leader_election.cc:290] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:51:38.269317 2066 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:38.269847 2066 raft_consensus.cc:695] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 1 LEADER]: Becoming Leader. State: Replica: bd9be859342c4cd98a85f651fc9789a3, State: Running, Role: LEADER
I20250622 13:51:38.270057 2063 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:51:38.270144 2066 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9be859342c4cd98a85f651fc9789a3" member_type: VOTER }
I20250622 13:51:38.271162 2067 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd9be859342c4cd98a85f651fc9789a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9be859342c4cd98a85f651fc9789a3" member_type: VOTER } }
I20250622 13:51:38.271245 2068 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd9be859342c4cd98a85f651fc9789a3. Latest consensus state: current_term: 1 leader_uuid: "bd9be859342c4cd98a85f651fc9789a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd9be859342c4cd98a85f651fc9789a3" member_type: VOTER } }
I20250622 13:51:38.271437 2067 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:38.271554 2068 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [sys.catalog]: This master's current role is: LEADER
I20250622 13:51:38.274017 2071 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:51:38.275564 2071 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:51:38.275992 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:51:38.278671 2071 catalog_manager.cc:1349] Generated new cluster ID: 5c2c6555b40244b99cb07a8f39167383
I20250622 13:51:38.278748 2071 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:51:38.288980 2071 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:51:38.289691 2071 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:51:38.296538 2071 catalog_manager.cc:5955] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3: Generated new TSK 0
I20250622 13:51:38.296747 2071 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:51:38.308856 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:38.311692 2084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:38.311745 2085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:38.312661 2087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:38.312983 413 server_base.cc:1048] running on GCE node
I20250622 13:51:38.313344 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:38.313460 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:38.313508 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600298313491 us; error 0 us; skew 500 ppm
I20250622 13:51:38.313638 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:38.314848 413 webserver.cc:469] Webserver started at http://127.0.103.65:41107/ using document root <none> and password file <none>
I20250622 13:51:38.315090 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:38.315156 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:38.315296 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:38.315891 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a"
format_stamp: "Formatted at 2025-06-22 13:51:38 on dist-test-slave-tmv0"
I20250622 13:51:38.318365 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250622 13:51:38.319911 2092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.320279 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:51:38.320402 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a"
format_stamp: "Formatted at 2025-06-22 13:51:38 on dist-test-slave-tmv0"
I20250622 13:51:38.320524 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:38.343134 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:38.343744 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:38.344471 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:38.345414 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:38.345510 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.345613 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:38.345695 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.360148 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:43969
I20250622 13:51:38.360158 2154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:43969 every 8 connection(s)
I20250622 13:51:38.362020 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:38.365332 2160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:38.365332 2159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:38.369052 2155 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41021
I20250622 13:51:38.369197 2155 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:38.369684 2155 heartbeater.cc:507] Master 127.0.103.126:41021 requested a full tablet report, sending...
W20250622 13:51:38.370934 2162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:38.371007 2028 ts_manager.cc:194] Registered new tserver with Master: b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65:43969)
I20250622 13:51:38.371130 413 server_base.cc:1048] running on GCE node
I20250622 13:51:38.372176 2028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43554
I20250622 13:51:38.372846 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:38.372941 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:38.373026 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600298372985 us; error 0 us; skew 500 ppm
I20250622 13:51:38.373184 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:38.374579 413 webserver.cc:469] Webserver started at http://127.0.103.66:43925/ using document root <none> and password file <none>
I20250622 13:51:38.374805 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:38.374912 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:38.375037 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:38.375649 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "bf2f64be4b15453f88dfee5c8cf646c1"
format_stamp: "Formatted at 2025-06-22 13:51:38 on dist-test-slave-tmv0"
I20250622 13:51:38.377992 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20250622 13:51:38.387106 2167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.387449 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:51:38.387549 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "bf2f64be4b15453f88dfee5c8cf646c1"
format_stamp: "Formatted at 2025-06-22 13:51:38 on dist-test-slave-tmv0"
I20250622 13:51:38.387655 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:38.399602 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:38.400095 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:38.400590 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:38.401309 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:38.401386 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.401469 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:38.401515 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.413838 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:41541
I20250622 13:51:38.413866 2229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:41541 every 8 connection(s)
I20250622 13:51:38.415330 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:38.418004 2233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:38.418311 2234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:38.418689 2236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:38.419411 413 server_base.cc:1048] running on GCE node
I20250622 13:51:38.419641 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:38.419692 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:38.419739 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600298419735 us; error 0 us; skew 500 ppm
I20250622 13:51:38.419911 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:38.421224 413 webserver.cc:469] Webserver started at http://127.0.103.67:41909/ using document root <none> and password file <none>
I20250622 13:51:38.421484 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:38.421561 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:38.421566 2230 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41021
I20250622 13:51:38.421685 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:38.421705 2230 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:38.421998 2230 heartbeater.cc:507] Master 127.0.103.126:41021 requested a full tablet report, sending...
I20250622 13:51:38.422317 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "313045cde1854fde99f49a63e173ac82"
format_stamp: "Formatted at 2025-06-22 13:51:38 on dist-test-slave-tmv0"
I20250622 13:51:38.422760 2028 ts_manager.cc:194] Registered new tserver with Master: bf2f64be4b15453f88dfee5c8cf646c1 (127.0.103.66:41541)
I20250622 13:51:38.423432 2028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43556
I20250622 13:51:38.424659 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:38.426019 2241 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.426334 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:51:38.426411 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "313045cde1854fde99f49a63e173ac82"
format_stamp: "Formatted at 2025-06-22 13:51:38 on dist-test-slave-tmv0"
I20250622 13:51:38.426501 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:38.444134 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:38.444684 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:38.445215 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:38.446017 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:38.446070 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.446135 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:38.446180 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:38.459946 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:34477
I20250622 13:51:38.459972 2303 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:34477 every 8 connection(s)
I20250622 13:51:38.466928 2304 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41021
I20250622 13:51:38.467051 2304 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:38.467311 2304 heartbeater.cc:507] Master 127.0.103.126:41021 requested a full tablet report, sending...
I20250622 13:51:38.468060 2028 ts_manager.cc:194] Registered new tserver with Master: 313045cde1854fde99f49a63e173ac82 (127.0.103.67:34477)
I20250622 13:51:38.468663 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007770698s
I20250622 13:51:38.468729 2028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43558
I20250622 13:51:39.373556 2155 heartbeater.cc:499] Master 127.0.103.126:41021 was elected leader, sending a full tablet report...
I20250622 13:51:39.424562 2230 heartbeater.cc:499] Master 127.0.103.126:41021 was elected leader, sending a full tablet report...
I20250622 13:51:39.470067 2304 heartbeater.cc:499] Master 127.0.103.126:41021 was elected leader, sending a full tablet report...
I20250622 13:51:39.496340 413 test_util.cc:276] Using random seed: 1694417805
I20250622 13:51:39.513731 2028 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43566:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:51:39.514492 2028 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:51:39.546555 2269 tablet_service.cc:1468] Processing CreateTablet for tablet d3be43d493e444c2b4321e50058dae6f (DEFAULT_TABLE table=test-workload [id=b22dc4fc4fa543238747bfef1e83c56c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250622 13:51:39.546661 2268 tablet_service.cc:1468] Processing CreateTablet for tablet 52aaeb6f8e694cb0ac1e4161946314d9 (DEFAULT_TABLE table=test-workload [id=b22dc4fc4fa543238747bfef1e83c56c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250622 13:51:39.547168 2269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3be43d493e444c2b4321e50058dae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:39.547312 2195 tablet_service.cc:1468] Processing CreateTablet for tablet d3be43d493e444c2b4321e50058dae6f (DEFAULT_TABLE table=test-workload [id=b22dc4fc4fa543238747bfef1e83c56c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250622 13:51:39.547395 2268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52aaeb6f8e694cb0ac1e4161946314d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:39.547641 2195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3be43d493e444c2b4321e50058dae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:39.553958 2194 tablet_service.cc:1468] Processing CreateTablet for tablet 52aaeb6f8e694cb0ac1e4161946314d9 (DEFAULT_TABLE table=test-workload [id=b22dc4fc4fa543238747bfef1e83c56c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250622 13:51:39.554638 2194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52aaeb6f8e694cb0ac1e4161946314d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:39.557942 2325 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1: Bootstrap starting.
I20250622 13:51:39.558349 2120 tablet_service.cc:1468] Processing CreateTablet for tablet d3be43d493e444c2b4321e50058dae6f (DEFAULT_TABLE table=test-workload [id=b22dc4fc4fa543238747bfef1e83c56c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250622 13:51:39.559013 2120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3be43d493e444c2b4321e50058dae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:39.559530 2325 tablet_bootstrap.cc:654] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:39.561851 2119 tablet_service.cc:1468] Processing CreateTablet for tablet 52aaeb6f8e694cb0ac1e4161946314d9 (DEFAULT_TABLE table=test-workload [id=b22dc4fc4fa543238747bfef1e83c56c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250622 13:51:39.562463 2119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52aaeb6f8e694cb0ac1e4161946314d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:39.570729 2324 tablet_bootstrap.cc:492] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82: Bootstrap starting.
I20250622 13:51:39.572651 2324 tablet_bootstrap.cc:654] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:39.573017 2325 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1: No bootstrap required, opened a new log
I20250622 13:51:39.573187 2325 ts_tablet_manager.cc:1397] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250622 13:51:39.575558 2327 tablet_bootstrap.cc:492] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Bootstrap starting.
I20250622 13:51:39.574160 2325 raft_consensus.cc:357] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.575744 2325 raft_consensus.cc:383] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:39.575867 2325 raft_consensus.cc:738] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2f64be4b15453f88dfee5c8cf646c1, State: Initialized, Role: FOLLOWER
I20250622 13:51:39.576130 2325 consensus_queue.cc:260] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.576809 2325 ts_tablet_manager.cc:1428] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:51:39.576938 2324 tablet_bootstrap.cc:492] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82: No bootstrap required, opened a new log
I20250622 13:51:39.577070 2325 tablet_bootstrap.cc:492] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1: Bootstrap starting.
I20250622 13:51:39.577108 2324 ts_tablet_manager.cc:1397] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250622 13:51:39.577936 2327 tablet_bootstrap.cc:654] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:39.577872 2324 raft_consensus.cc:357] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.578186 2324 raft_consensus.cc:383] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:39.578289 2324 raft_consensus.cc:738] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313045cde1854fde99f49a63e173ac82, State: Initialized, Role: FOLLOWER
I20250622 13:51:39.578657 2325 tablet_bootstrap.cc:654] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:39.578608 2324 consensus_queue.cc:260] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.583070 2324 ts_tablet_manager.cc:1428] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20250622 13:51:39.583353 2324 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82: Bootstrap starting.
I20250622 13:51:39.583406 2325 tablet_bootstrap.cc:492] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1: No bootstrap required, opened a new log
I20250622 13:51:39.583544 2325 ts_tablet_manager.cc:1397] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250622 13:51:39.584009 2327 tablet_bootstrap.cc:492] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a: No bootstrap required, opened a new log
I20250622 13:51:39.584128 2327 ts_tablet_manager.cc:1397] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20250622 13:51:39.584280 2325 raft_consensus.cc:357] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.584460 2325 raft_consensus.cc:383] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:39.584517 2325 raft_consensus.cc:738] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2f64be4b15453f88dfee5c8cf646c1, State: Initialized, Role: FOLLOWER
I20250622 13:51:39.584722 2325 consensus_queue.cc:260] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.585327 2325 ts_tablet_manager.cc:1428] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:51:39.585806 2324 tablet_bootstrap.cc:654] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:39.585952 2327 raft_consensus.cc:357] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.586279 2327 raft_consensus.cc:383] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:39.587217 2327 raft_consensus.cc:738] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a434f8a43a4bb2aa87f6f0f2d83a2a, State: Initialized, Role: FOLLOWER
I20250622 13:51:39.587519 2324 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82: No bootstrap required, opened a new log
I20250622 13:51:39.587630 2324 ts_tablet_manager.cc:1397] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:51:39.587675 2327 consensus_queue.cc:260] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.588356 2324 raft_consensus.cc:357] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.588537 2324 raft_consensus.cc:383] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:39.588604 2324 raft_consensus.cc:738] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313045cde1854fde99f49a63e173ac82, State: Initialized, Role: FOLLOWER
I20250622 13:51:39.588881 2324 consensus_queue.cc:260] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.589491 2324 ts_tablet_manager.cc:1428] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:51:39.589859 2327 ts_tablet_manager.cc:1428] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20250622 13:51:39.590135 2327 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Bootstrap starting.
I20250622 13:51:39.591706 2327 tablet_bootstrap.cc:654] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Neither blocks nor log segments found. Creating new log.
I20250622 13:51:39.593678 2327 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: No bootstrap required, opened a new log
I20250622 13:51:39.593770 2327 ts_tablet_manager.cc:1397] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250622 13:51:39.594218 2327 raft_consensus.cc:357] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.594333 2327 raft_consensus.cc:383] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:51:39.594365 2327 raft_consensus.cc:738] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a434f8a43a4bb2aa87f6f0f2d83a2a, State: Initialized, Role: FOLLOWER
I20250622 13:51:39.594540 2327 consensus_queue.cc:260] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.594908 2327 ts_tablet_manager.cc:1428] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250622 13:51:39.638096 2334 raft_consensus.cc:491] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:39.638262 2334 raft_consensus.cc:513] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.638924 2334 leader_election.cc:290] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2f64be4b15453f88dfee5c8cf646c1 (127.0.103.66:41541), 313045cde1854fde99f49a63e173ac82 (127.0.103.67:34477)
I20250622 13:51:39.646015 2205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52aaeb6f8e694cb0ac1e4161946314d9" candidate_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" is_pre_election: true
I20250622 13:51:39.646015 2279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52aaeb6f8e694cb0ac1e4161946314d9" candidate_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313045cde1854fde99f49a63e173ac82" is_pre_election: true
I20250622 13:51:39.646332 2279 raft_consensus.cc:2466] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1a434f8a43a4bb2aa87f6f0f2d83a2a in term 0.
I20250622 13:51:39.646332 2205 raft_consensus.cc:2466] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1a434f8a43a4bb2aa87f6f0f2d83a2a in term 0.
I20250622 13:51:39.646799 2093 leader_election.cc:304] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1a434f8a43a4bb2aa87f6f0f2d83a2a, bf2f64be4b15453f88dfee5c8cf646c1; no voters:
I20250622 13:51:39.647051 2334 raft_consensus.cc:2802] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:39.647135 2334 raft_consensus.cc:491] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:39.647212 2334 raft_consensus.cc:3058] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:39.648905 2334 raft_consensus.cc:513] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.649435 2334 leader_election.cc:290] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [CANDIDATE]: Term 1 election: Requested vote from peers bf2f64be4b15453f88dfee5c8cf646c1 (127.0.103.66:41541), 313045cde1854fde99f49a63e173ac82 (127.0.103.67:34477)
I20250622 13:51:39.649775 2205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52aaeb6f8e694cb0ac1e4161946314d9" candidate_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2f64be4b15453f88dfee5c8cf646c1"
I20250622 13:51:39.649822 2279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52aaeb6f8e694cb0ac1e4161946314d9" candidate_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313045cde1854fde99f49a63e173ac82"
I20250622 13:51:39.649935 2279 raft_consensus.cc:3058] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:39.649962 2205 raft_consensus.cc:3058] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:39.651656 2279 raft_consensus.cc:2466] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1a434f8a43a4bb2aa87f6f0f2d83a2a in term 1.
I20250622 13:51:39.651875 2205 raft_consensus.cc:2466] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1a434f8a43a4bb2aa87f6f0f2d83a2a in term 1.
I20250622 13:51:39.651971 2096 leader_election.cc:304] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313045cde1854fde99f49a63e173ac82, b1a434f8a43a4bb2aa87f6f0f2d83a2a; no voters:
I20250622 13:51:39.652238 2334 raft_consensus.cc:2802] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:39.652612 2334 raft_consensus.cc:695] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 LEADER]: Becoming Leader. State: Replica: b1a434f8a43a4bb2aa87f6f0f2d83a2a, State: Running, Role: LEADER
I20250622 13:51:39.652868 2334 consensus_queue.cc:237] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } }
I20250622 13:51:39.655323 2028 catalog_manager.cc:5582] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a reported cstate change: term changed from 0 to 1, leader changed from <none> to b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:39.662488 2329 raft_consensus.cc:491] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:51:39.662616 2329 raft_consensus.cc:513] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.663291 2329 leader_election.cc:290] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 313045cde1854fde99f49a63e173ac82 (127.0.103.67:34477), b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65:43969)
I20250622 13:51:39.669661 2279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3be43d493e444c2b4321e50058dae6f" candidate_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313045cde1854fde99f49a63e173ac82" is_pre_election: true
I20250622 13:51:39.669693 2130 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3be43d493e444c2b4321e50058dae6f" candidate_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" is_pre_election: true
I20250622 13:51:39.669848 2279 raft_consensus.cc:2466] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf2f64be4b15453f88dfee5c8cf646c1 in term 0.
I20250622 13:51:39.669885 2130 raft_consensus.cc:2466] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf2f64be4b15453f88dfee5c8cf646c1 in term 0.
I20250622 13:51:39.670289 2169 leader_election.cc:304] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 313045cde1854fde99f49a63e173ac82, b1a434f8a43a4bb2aa87f6f0f2d83a2a, bf2f64be4b15453f88dfee5c8cf646c1; no voters:
I20250622 13:51:39.670557 2329 raft_consensus.cc:2802] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:51:39.670632 2329 raft_consensus.cc:491] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:51:39.670714 2329 raft_consensus.cc:3058] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:39.672508 2329 raft_consensus.cc:513] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.672960 2329 leader_election.cc:290] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [CANDIDATE]: Term 1 election: Requested vote from peers 313045cde1854fde99f49a63e173ac82 (127.0.103.67:34477), b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65:43969)
I20250622 13:51:39.673367 2279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3be43d493e444c2b4321e50058dae6f" candidate_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313045cde1854fde99f49a63e173ac82"
I20250622 13:51:39.673424 2130 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3be43d493e444c2b4321e50058dae6f" candidate_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a"
I20250622 13:51:39.673588 2130 raft_consensus.cc:3058] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:39.673596 2279 raft_consensus.cc:3058] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:51:39.676610 2130 raft_consensus.cc:2466] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf2f64be4b15453f88dfee5c8cf646c1 in term 1.
I20250622 13:51:39.676587 2279 raft_consensus.cc:2466] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf2f64be4b15453f88dfee5c8cf646c1 in term 1.
I20250622 13:51:39.677012 2169 leader_election.cc:304] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1a434f8a43a4bb2aa87f6f0f2d83a2a, bf2f64be4b15453f88dfee5c8cf646c1; no voters:
I20250622 13:51:39.677292 2329 raft_consensus.cc:2802] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:51:39.677717 2329 raft_consensus.cc:695] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 LEADER]: Becoming Leader. State: Replica: bf2f64be4b15453f88dfee5c8cf646c1, State: Running, Role: LEADER
I20250622 13:51:39.677966 2329 consensus_queue.cc:237] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } }
I20250622 13:51:39.679991 2028 catalog_manager.cc:5582] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf2f64be4b15453f88dfee5c8cf646c1 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } health_report { overall_health: UNKNOWN } } }
I20250622 13:51:40.033561 2334 consensus_queue.cc:1035] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:40.038193 2338 consensus_queue.cc:1035] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:40.128978 2329 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:40.132988 2342 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:51:40.275611 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:40.276191 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:40.276351 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:42.276791 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:42.277321 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:42.277500 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:44.277988 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:44.278525 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:44.278645 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:46.279028 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:46.279597 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:46.279699 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:48.280131 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:48.280678 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:48.280805 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:50.281195 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:50.281823 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:50.281935 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:50.757308 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:51:50.760356 2362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:51:50.760478 2361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:50.762902 413 server_base.cc:1048] running on GCE node
W20250622 13:51:50.763046 2364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:51:50.763440 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:51:50.763518 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:51:50.763593 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600310763561 us; error 0 us; skew 500 ppm
I20250622 13:51:50.763774 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:51:50.771148 413 webserver.cc:469] Webserver started at http://127.0.103.68:44329/ using document root <none> and password file <none>
I20250622 13:51:50.771389 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:51:50.771472 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:51:50.771584 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:51:50.772173 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-3-root/instance:
uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4"
format_stamp: "Formatted at 2025-06-22 13:51:50 on dist-test-slave-tmv0"
I20250622 13:51:50.774704 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:51:50.776194 2369 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:50.776554 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:51:50.776669 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-3-root
uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4"
format_stamp: "Formatted at 2025-06-22 13:51:50 on dist-test-slave-tmv0"
I20250622 13:51:50.776798 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:51:50.790359 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:51:50.790951 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:51:50.791519 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:51:50.792318 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:51:50.792390 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:50.792505 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:51:50.792562 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:51:50.808419 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:38821
I20250622 13:51:50.808439 2431 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:38821 every 8 connection(s)
I20250622 13:51:50.815567 2432 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41021
I20250622 13:51:50.815724 2432 heartbeater.cc:461] Registering TS with master...
I20250622 13:51:50.816107 2432 heartbeater.cc:507] Master 127.0.103.126:41021 requested a full tablet report, sending...
I20250622 13:51:50.816907 2028 ts_manager.cc:194] Registered new tserver with Master: 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68:38821)
I20250622 13:51:50.817763 2028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57702
I20250622 13:51:51.297405 2205 consensus_queue.cc:237] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: true } }
I20250622 13:51:51.300498 2278 raft_consensus.cc:1273] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Refusing update from remote peer bf2f64be4b15453f88dfee5c8cf646c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:51.301012 2130 raft_consensus.cc:1273] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Refusing update from remote peer bf2f64be4b15453f88dfee5c8cf646c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:51:51.301110 2439 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:51.301666 2440 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:51:51.304024 2439 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: true } } }
I20250622 13:51:51.304669 2278 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: true } } }
I20250622 13:51:51.305043 2130 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: true } } }
I20250622 13:51:51.309893 2028 catalog_manager.cc:5582] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a reported cstate change: config changed from index -1 to 2, NON_VOTER 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: true } } }
W20250622 13:51:51.314831 2171 consensus_peers.cc:487] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 -> Peer 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68:38821): Couldn't send request to peer 62ae305e2ffa4ad3b44736b24f0ee1c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3be43d493e444c2b4321e50058dae6f. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:51:51.779379 2450 ts_tablet_manager.cc:927] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: Initiating tablet copy from peer bf2f64be4b15453f88dfee5c8cf646c1 (127.0.103.66:41541)
I20250622 13:51:51.780331 2450 tablet_copy_client.cc:323] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:41541
I20250622 13:51:51.785892 2215 tablet_copy_service.cc:140] P bf2f64be4b15453f88dfee5c8cf646c1: Received BeginTabletCopySession request for tablet d3be43d493e444c2b4321e50058dae6f from peer 62ae305e2ffa4ad3b44736b24f0ee1c4 ({username='slave'} at 127.0.0.1:55642)
I20250622 13:51:51.786054 2215 tablet_copy_service.cc:161] P bf2f64be4b15453f88dfee5c8cf646c1: Beginning new tablet copy session on tablet d3be43d493e444c2b4321e50058dae6f from peer 62ae305e2ffa4ad3b44736b24f0ee1c4 at {username='slave'} at 127.0.0.1:55642: session id = 62ae305e2ffa4ad3b44736b24f0ee1c4-d3be43d493e444c2b4321e50058dae6f
I20250622 13:51:51.789006 2215 tablet_copy_source_session.cc:215] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:51:51.791333 2450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3be43d493e444c2b4321e50058dae6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:51:51.797281 2450 tablet_copy_client.cc:806] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: tablet copy: Starting download of 0 data blocks...
I20250622 13:51:51.797592 2450 tablet_copy_client.cc:670] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: tablet copy: Starting download of 1 WAL segments...
I20250622 13:51:51.799531 2450 tablet_copy_client.cc:538] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:51:51.802084 2450 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: Bootstrap starting.
I20250622 13:51:51.807873 2450 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:51:51.808236 2450 tablet_bootstrap.cc:492] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: Bootstrap complete.
I20250622 13:51:51.808447 2450 ts_tablet_manager.cc:1397] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20250622 13:51:51.809178 2450 raft_consensus.cc:357] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: true } }
I20250622 13:51:51.809381 2450 raft_consensus.cc:738] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 62ae305e2ffa4ad3b44736b24f0ee1c4, State: Initialized, Role: LEARNER
I20250622 13:51:51.809569 2450 consensus_queue.cc:260] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: true } }
I20250622 13:51:51.810249 2450 ts_tablet_manager.cc:1428] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250622 13:51:51.810279 2432 heartbeater.cc:499] Master 127.0.103.126:41021 was elected leader, sending a full tablet report...
I20250622 13:51:51.810922 2215 tablet_copy_service.cc:342] P bf2f64be4b15453f88dfee5c8cf646c1: Request end of tablet copy session 62ae305e2ffa4ad3b44736b24f0ee1c4-d3be43d493e444c2b4321e50058dae6f received from {username='slave'} at 127.0.0.1:55642
I20250622 13:51:51.811045 2215 tablet_copy_service.cc:434] P bf2f64be4b15453f88dfee5c8cf646c1: ending tablet copy session 62ae305e2ffa4ad3b44736b24f0ee1c4-d3be43d493e444c2b4321e50058dae6f on tablet d3be43d493e444c2b4321e50058dae6f with peer 62ae305e2ffa4ad3b44736b24f0ee1c4
I20250622 13:51:52.282202 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:52.282773 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:52.282997 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:52.367257 2407 raft_consensus.cc:1215] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250622 13:51:52.987974 2455 raft_consensus.cc:1062] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1: attempting to promote NON_VOTER 62ae305e2ffa4ad3b44736b24f0ee1c4 to VOTER
I20250622 13:51:52.989249 2455 consensus_queue.cc:237] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } }
I20250622 13:51:52.991879 2130 raft_consensus.cc:1273] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Refusing update from remote peer bf2f64be4b15453f88dfee5c8cf646c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:51:52.991930 2407 raft_consensus.cc:1273] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 LEARNER]: Refusing update from remote peer bf2f64be4b15453f88dfee5c8cf646c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:51:52.992419 2458 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:51:52.993126 2458 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:51:52.996234 2458 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } } }
I20250622 13:51:52.997226 2407 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } } }
I20250622 13:51:53.002311 2027 catalog_manager.cc:5582] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 reported cstate change: config changed from index 2 to 3, 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } } }
I20250622 13:51:53.005204 2278 raft_consensus.cc:1273] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Refusing update from remote peer bf2f64be4b15453f88dfee5c8cf646c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:51:53.005198 2130 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } } }
I20250622 13:51:53.006455 2464 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:51:53.011986 2278 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 62ae305e2ffa4ad3b44736b24f0ee1c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "b1a434f8a43a4bb2aa87f6f0f2d83a2a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43969 } attrs { replace: true } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } } }
I20250622 13:51:53.585481 2204 consensus_queue.cc:237] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } }
I20250622 13:51:53.587211 2407 raft_consensus.cc:1273] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 FOLLOWER]: Refusing update from remote peer bf2f64be4b15453f88dfee5c8cf646c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250622 13:51:53.587220 2278 raft_consensus.cc:1273] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Refusing update from remote peer bf2f64be4b15453f88dfee5c8cf646c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250622 13:51:53.587702 2472 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:51:53.588003 2471 consensus_queue.cc:1035] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:51:53.590812 2471 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } } }
I20250622 13:51:53.593770 2407 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } } }
I20250622 13:51:53.593524 2012 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d3be43d493e444c2b4321e50058dae6f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:51:53.594786 2026 catalog_manager.cc:5582] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 reported cstate change: config changed from index 3 to 4, VOTER b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:51:53.595342 2278 raft_consensus.cc:2953] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "313045cde1854fde99f49a63e173ac82" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34477 } } peers { permanent_uuid: "bf2f64be4b15453f88dfee5c8cf646c1" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41541 } } peers { permanent_uuid: "62ae305e2ffa4ad3b44736b24f0ee1c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38821 } attrs { promote: false } } }
I20250622 13:51:53.603224 2119 tablet_service.cc:1515] Processing DeleteTablet for tablet d3be43d493e444c2b4321e50058dae6f with delete_type TABLET_DATA_TOMBSTONED (TS b1a434f8a43a4bb2aa87f6f0f2d83a2a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59014
I20250622 13:51:53.604249 2478 tablet_replica.cc:331] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: stopping tablet replica
I20250622 13:51:53.604451 2478 raft_consensus.cc:2241] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:51:53.604750 2478 raft_consensus.cc:2270] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:51:53.605595 2478 ts_tablet_manager.cc:1905] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:51:53.610198 2478 ts_tablet_manager.cc:1918] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250622 13:51:53.610311 2478 log.cc:1199] T d3be43d493e444c2b4321e50058dae6f P b1a434f8a43a4bb2aa87f6f0f2d83a2a: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750600281110842-413-0/minicluster-data/ts-0-root/wals/d3be43d493e444c2b4321e50058dae6f
I20250622 13:51:53.610893 2013 catalog_manager.cc:4928] TS b1a434f8a43a4bb2aa87f6f0f2d83a2a (127.0.103.65:43969): tablet d3be43d493e444c2b4321e50058dae6f (table test-workload [id=b22dc4fc4fa543238747bfef1e83c56c]) successfully deleted
I20250622 13:51:54.283428 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:54.284029 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:54.284130 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:56.284546 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:56.285148 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:56.285261 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:51:58.285698 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:51:58.286266 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:51:58.286371 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:00.286921 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:00.287487 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:00.287583 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:02.288053 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:02.288718 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:02.288842 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:04.289204 2081 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:04.289846 2081 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:04.289978 2081 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:04.890923 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:52:04.900872 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:04.901191 413 tablet_replica.cc:331] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a: stopping tablet replica
I20250622 13:52:04.901382 413 raft_consensus.cc:2241] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:04.901705 413 raft_consensus.cc:2270] T 52aaeb6f8e694cb0ac1e4161946314d9 P b1a434f8a43a4bb2aa87f6f0f2d83a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:04.918247 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:52:04.921267 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:52:04.936095 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:04.936406 413 tablet_replica.cc:331] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1: stopping tablet replica
I20250622 13:52:04.936615 413 raft_consensus.cc:2241] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:52:04.936774 413 raft_consensus.cc:2270] T 52aaeb6f8e694cb0ac1e4161946314d9 P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:04.937306 413 tablet_replica.cc:331] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1: stopping tablet replica
I20250622 13:52:04.937448 413 raft_consensus.cc:2241] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:04.937698 413 raft_consensus.cc:2270] T d3be43d493e444c2b4321e50058dae6f P bf2f64be4b15453f88dfee5c8cf646c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:04.951591 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:52:04.954643 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:52:04.963207 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:04.963480 413 tablet_replica.cc:331] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82: stopping tablet replica
I20250622 13:52:04.963670 413 raft_consensus.cc:2241] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:52:04.963894 413 raft_consensus.cc:2270] T d3be43d493e444c2b4321e50058dae6f P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:04.964407 413 tablet_replica.cc:331] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82: stopping tablet replica
I20250622 13:52:04.964506 413 raft_consensus.cc:2241] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:52:04.964609 413 raft_consensus.cc:2270] T 52aaeb6f8e694cb0ac1e4161946314d9 P 313045cde1854fde99f49a63e173ac82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:04.978114 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:52:04.980893 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250622 13:52:04.988399 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:04.988646 413 tablet_replica.cc:331] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4: stopping tablet replica
I20250622 13:52:04.988802 413 raft_consensus.cc:2241] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:52:04.989012 413 raft_consensus.cc:2270] T d3be43d493e444c2b4321e50058dae6f P 62ae305e2ffa4ad3b44736b24f0ee1c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:05.002326 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250622 13:52:05.005071 413 master.cc:561] Master@127.0.103.126:41021 shutting down...
I20250622 13:52:05.012521 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:05.012739 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:05.012837 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P bd9be859342c4cd98a85f651fc9789a3: stopping tablet replica
I20250622 13:52:05.026786 413 master.cc:583] Master@127.0.103.126:41021 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26834 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20250622 13:52:05.038569 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:41963
I20250622 13:52:05.039067 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:05.041535 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:05.041685 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:05.041814 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:05.042070 413 server_base.cc:1048] running on GCE node
I20250622 13:52:05.042287 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:05.042335 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:05.042376 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600325042372 us; error 0 us; skew 500 ppm
I20250622 13:52:05.042516 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:05.043732 413 webserver.cc:469] Webserver started at http://127.0.103.126:36369/ using document root <none> and password file <none>
I20250622 13:52:05.043931 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:05.043998 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:05.044131 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:05.044692 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3"
format_stamp: "Formatted at 2025-06-22 13:52:05 on dist-test-slave-tmv0"
I20250622 13:52:05.047044 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:05.048406 2490 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:05.048717 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:05.048815 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3"
format_stamp: "Formatted at 2025-06-22 13:52:05 on dist-test-slave-tmv0"
I20250622 13:52:05.048914 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:05.057775 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:05.058261 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:05.069248 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:41963
I20250622 13:52:05.069270 2541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:41963 every 8 connection(s)
I20250622 13:52:05.070461 2542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:05.074329 2542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3: Bootstrap starting.
I20250622 13:52:05.075429 2542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:05.076746 2542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3: No bootstrap required, opened a new log
I20250622 13:52:05.077304 2542 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3" member_type: VOTER }
I20250622 13:52:05.077468 2542 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:05.077529 2542 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b17fb0d24424a8dbf6e294cb9ac23d3, State: Initialized, Role: FOLLOWER
I20250622 13:52:05.077705 2542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3" member_type: VOTER }
I20250622 13:52:05.077802 2542 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:52:05.077855 2542 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:52:05.077934 2542 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:05.079545 2542 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3" member_type: VOTER }
I20250622 13:52:05.079708 2542 leader_election.cc:304] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b17fb0d24424a8dbf6e294cb9ac23d3; no voters:
I20250622 13:52:05.080101 2542 leader_election.cc:290] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:52:05.080204 2545 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:05.080703 2545 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2b17fb0d24424a8dbf6e294cb9ac23d3, State: Running, Role: LEADER
I20250622 13:52:05.080912 2545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3" member_type: VOTER }
I20250622 13:52:05.080981 2542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:52:05.081700 2547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b17fb0d24424a8dbf6e294cb9ac23d3. Latest consensus state: current_term: 1 leader_uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3" member_type: VOTER } }
I20250622 13:52:05.081854 2547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:05.081887 2546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b17fb0d24424a8dbf6e294cb9ac23d3" member_type: VOTER } }
I20250622 13:52:05.082038 2546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:05.085511 2550 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:52:05.086658 2550 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:52:05.089465 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:52:05.089466 2550 catalog_manager.cc:1349] Generated new cluster ID: 1c2a995194514343af55662eec91304a
I20250622 13:52:05.089576 2550 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:52:05.104197 2550 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:52:05.104946 2550 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:52:05.113547 2550 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3: Generated new TSK 0
I20250622 13:52:05.113770 2550 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:52:05.121659 413 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000100829s
I20250622 13:52:17.209030 413 master.cc:561] Master@127.0.103.126:41963 shutting down...
I20250622 13:52:17.226526 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:17.226756 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:17.226840 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2b17fb0d24424a8dbf6e294cb9ac23d3: stopping tablet replica
I20250622 13:52:17.249470 413 master.cc:583] Master@127.0.103.126:41963 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12219 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20250622 13:52:17.257675 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:42649
I20250622 13:52:17.258167 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:17.260586 2564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:17.260740 2565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:17.260934 2567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:17.261206 413 server_base.cc:1048] running on GCE node
I20250622 13:52:17.261972 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:17.262045 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:17.262099 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600337262078 us; error 0 us; skew 500 ppm
I20250622 13:52:17.262264 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:17.263537 413 webserver.cc:469] Webserver started at http://127.0.103.126:46379/ using document root <none> and password file <none>
I20250622 13:52:17.263760 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:17.263839 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:17.263988 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:17.264606 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "841cb375436f4131884b198da31020bb"
format_stamp: "Formatted at 2025-06-22 13:52:17 on dist-test-slave-tmv0"
I20250622 13:52:17.267102 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:17.268699 2572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.269063 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:17.269181 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "841cb375436f4131884b198da31020bb"
format_stamp: "Formatted at 2025-06-22 13:52:17 on dist-test-slave-tmv0"
I20250622 13:52:17.269302 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:17.283941 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:17.284533 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:17.296794 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:42649
I20250622 13:52:17.296823 2623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:42649 every 8 connection(s)
I20250622 13:52:17.298178 2624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:17.302311 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb: Bootstrap starting.
I20250622 13:52:17.303467 2624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:17.304891 2624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb: No bootstrap required, opened a new log
I20250622 13:52:17.305575 2624 raft_consensus.cc:357] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841cb375436f4131884b198da31020bb" member_type: VOTER }
I20250622 13:52:17.305724 2624 raft_consensus.cc:383] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:17.305780 2624 raft_consensus.cc:738] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841cb375436f4131884b198da31020bb, State: Initialized, Role: FOLLOWER
I20250622 13:52:17.305975 2624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841cb375436f4131884b198da31020bb" member_type: VOTER }
I20250622 13:52:17.306095 2624 raft_consensus.cc:397] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:52:17.306150 2624 raft_consensus.cc:491] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:52:17.306216 2624 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:17.307957 2624 raft_consensus.cc:513] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841cb375436f4131884b198da31020bb" member_type: VOTER }
I20250622 13:52:17.308131 2624 leader_election.cc:304] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 841cb375436f4131884b198da31020bb; no voters:
I20250622 13:52:17.308503 2624 leader_election.cc:290] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:52:17.308624 2627 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:17.309144 2627 raft_consensus.cc:695] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 1 LEADER]: Becoming Leader. State: Replica: 841cb375436f4131884b198da31020bb, State: Running, Role: LEADER
I20250622 13:52:17.309415 2624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:52:17.309456 2627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841cb375436f4131884b198da31020bb" member_type: VOTER }
I20250622 13:52:17.310364 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "841cb375436f4131884b198da31020bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841cb375436f4131884b198da31020bb" member_type: VOTER } }
I20250622 13:52:17.310526 2629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841cb375436f4131884b198da31020bb. Latest consensus state: current_term: 1 leader_uuid: "841cb375436f4131884b198da31020bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841cb375436f4131884b198da31020bb" member_type: VOTER } }
I20250622 13:52:17.310782 2629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:17.310781 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:17.312956 2633 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:52:17.315033 2633 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:52:17.316514 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:52:17.318373 2633 catalog_manager.cc:1349] Generated new cluster ID: c4a6c60d30a54477ab3e5e2b50f18565
I20250622 13:52:17.318446 2633 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:52:17.331486 2633 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:52:17.332295 2633 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:52:17.343413 2633 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb: Generated new TSK 0
I20250622 13:52:17.343688 2633 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:52:17.349202 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:17.351819 2645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:17.351984 2646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:17.352442 2648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:17.352766 413 server_base.cc:1048] running on GCE node
I20250622 13:52:17.352986 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:17.353051 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:17.353117 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600337353099 us; error 0 us; skew 500 ppm
I20250622 13:52:17.353264 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:17.354466 413 webserver.cc:469] Webserver started at http://127.0.103.65:37395/ using document root <none> and password file <none>
I20250622 13:52:17.354700 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:17.354780 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:17.354924 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:17.355584 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "06164031eaf24e3d833f0f3dfd1dfb8a"
format_stamp: "Formatted at 2025-06-22 13:52:17 on dist-test-slave-tmv0"
I20250622 13:52:17.357972 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:17.359426 2653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.359764 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:17.359872 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "06164031eaf24e3d833f0f3dfd1dfb8a"
format_stamp: "Formatted at 2025-06-22 13:52:17 on dist-test-slave-tmv0"
I20250622 13:52:17.359994 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:17.366395 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:17.366927 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:17.367384 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:17.368140 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:17.368208 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.368306 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:17.368384 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.381105 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45185
I20250622 13:52:17.381125 2715 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:45185 every 8 connection(s)
I20250622 13:52:17.382967 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:17.386111 2720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:17.386420 2721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:17.387539 2723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:17.389027 413 server_base.cc:1048] running on GCE node
I20250622 13:52:17.389684 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:17.389757 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:17.389817 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600337389795 us; error 0 us; skew 500 ppm
I20250622 13:52:17.389981 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:17.391287 2716 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42649
I20250622 13:52:17.391330 413 webserver.cc:469] Webserver started at http://127.0.103.66:42405/ using document root <none> and password file <none>
I20250622 13:52:17.391458 2716 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:17.391603 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:17.391700 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:17.391848 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:17.391903 2716 heartbeater.cc:507] Master 127.0.103.126:42649 requested a full tablet report, sending...
I20250622 13:52:17.392597 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "55a69c857bf94d24a8c0a8e9875c35c7"
format_stamp: "Formatted at 2025-06-22 13:52:17 on dist-test-slave-tmv0"
I20250622 13:52:17.392791 2589 ts_manager.cc:194] Registered new tserver with Master: 06164031eaf24e3d833f0f3dfd1dfb8a (127.0.103.65:45185)
I20250622 13:52:17.393880 2589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48014
I20250622 13:52:17.395258 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:17.396739 2728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.397063 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:17.397176 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "55a69c857bf94d24a8c0a8e9875c35c7"
format_stamp: "Formatted at 2025-06-22 13:52:17 on dist-test-slave-tmv0"
I20250622 13:52:17.397310 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:17.403805 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:17.404307 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:17.404860 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:17.405678 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:17.405752 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.405853 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:17.405913 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.418919 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:35995
I20250622 13:52:17.418960 2790 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:35995 every 8 connection(s)
I20250622 13:52:17.420589 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:17.423223 2794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:17.424222 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:17.424829 2795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:17.426599 413 server_base.cc:1048] running on GCE node
I20250622 13:52:17.426940 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:17.427019 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:17.427071 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600337427046 us; error 0 us; skew 500 ppm
I20250622 13:52:17.427238 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:17.428488 2791 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42649
I20250622 13:52:17.428704 413 webserver.cc:469] Webserver started at http://127.0.103.67:46475/ using document root <none> and password file <none>
I20250622 13:52:17.428730 2791 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:17.428965 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:17.429056 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:17.429158 2791 heartbeater.cc:507] Master 127.0.103.126:42649 requested a full tablet report, sending...
I20250622 13:52:17.429189 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:17.429996 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "e33a1bbe24554d3d9af3a5d586e6648b"
format_stamp: "Formatted at 2025-06-22 13:52:17 on dist-test-slave-tmv0"
I20250622 13:52:17.430080 2589 ts_manager.cc:194] Registered new tserver with Master: 55a69c857bf94d24a8c0a8e9875c35c7 (127.0.103.66:35995)
I20250622 13:52:17.430837 2589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48026
I20250622 13:52:17.432524 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:17.433928 2802 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.434211 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:17.434315 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "e33a1bbe24554d3d9af3a5d586e6648b"
format_stamp: "Formatted at 2025-06-22 13:52:17 on dist-test-slave-tmv0"
I20250622 13:52:17.434424 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:17.447793 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:17.448261 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:17.448755 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:17.449544 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:17.449621 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.449733 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:17.449801 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:17.462924 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:46203
I20250622 13:52:17.462953 2864 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:46203 every 8 connection(s)
I20250622 13:52:17.469602 2865 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42649
I20250622 13:52:17.469727 2865 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:17.470029 2865 heartbeater.cc:507] Master 127.0.103.126:42649 requested a full tablet report, sending...
I20250622 13:52:17.470796 2589 ts_manager.cc:194] Registered new tserver with Master: e33a1bbe24554d3d9af3a5d586e6648b (127.0.103.67:46203)
I20250622 13:52:17.471504 2589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48038
I20250622 13:52:17.471743 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007947304s
I20250622 13:52:18.395186 2716 heartbeater.cc:499] Master 127.0.103.126:42649 was elected leader, sending a full tablet report...
I20250622 13:52:18.432077 2791 heartbeater.cc:499] Master 127.0.103.126:42649 was elected leader, sending a full tablet report...
I20250622 13:52:18.472888 2865 heartbeater.cc:499] Master 127.0.103.126:42649 was elected leader, sending a full tablet report...
I20250622 13:52:19.316483 2642 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:21.316913 2642 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:23.317652 2642 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:25.318210 2642 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:27.318768 2642 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:29.319489 2642 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:29.554747 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:52:29.567314 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:29.580658 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:52:29.583413 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:52:29.591598 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:29.605850 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:52:29.608453 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:52:29.617065 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:29.629935 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:52:29.642097 413 master.cc:561] Master@127.0.103.126:42649 shutting down...
I20250622 13:52:29.649704 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:29.649955 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:29.650063 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 841cb375436f4131884b198da31020bb: stopping tablet replica
I20250622 13:52:29.663389 413 master.cc:583] Master@127.0.103.126:42649 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12414 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20250622 13:52:29.672124 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:41247
I20250622 13:52:29.672614 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:29.674993 2869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:29.675252 2870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:29.675402 2872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:29.675740 413 server_base.cc:1048] running on GCE node
I20250622 13:52:29.675963 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:29.676017 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:29.676047 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600349676045 us; error 0 us; skew 500 ppm
I20250622 13:52:29.676168 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:29.677506 413 webserver.cc:469] Webserver started at http://127.0.103.126:40875/ using document root <none> and password file <none>
I20250622 13:52:29.677711 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:29.677774 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:29.677866 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:29.678521 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.681119 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:29.682904 2877 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.683264 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:29.683365 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.683506 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:29.694311 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:29.694855 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:29.705673 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:41247
I20250622 13:52:29.705693 2928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:41247 every 8 connection(s)
I20250622 13:52:29.706879 2929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:29.710649 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4: Bootstrap starting.
I20250622 13:52:29.711700 2929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:29.713037 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4: No bootstrap required, opened a new log
I20250622 13:52:29.713608 2929 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4" member_type: VOTER }
I20250622 13:52:29.713723 2929 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:29.713761 2929 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6fd06a3d504d75a029c6a67cf4c2a4, State: Initialized, Role: FOLLOWER
I20250622 13:52:29.713891 2929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4" member_type: VOTER }
I20250622 13:52:29.714015 2929 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:52:29.714116 2929 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:52:29.714233 2929 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:29.716491 2929 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4" member_type: VOTER }
I20250622 13:52:29.716686 2929 leader_election.cc:304] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b6fd06a3d504d75a029c6a67cf4c2a4; no voters:
I20250622 13:52:29.717043 2929 leader_election.cc:290] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:52:29.717147 2932 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:29.717636 2932 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b6fd06a3d504d75a029c6a67cf4c2a4, State: Running, Role: LEADER
I20250622 13:52:29.717801 2929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:52:29.717902 2932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4" member_type: VOTER }
I20250622 13:52:29.719023 2934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6fd06a3d504d75a029c6a67cf4c2a4. Latest consensus state: current_term: 1 leader_uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4" member_type: VOTER } }
I20250622 13:52:29.719208 2934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:29.724085 2933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6fd06a3d504d75a029c6a67cf4c2a4" member_type: VOTER } }
I20250622 13:52:29.724252 2933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:29.724282 2940 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:52:29.725526 2940 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:52:29.727519 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:52:29.728303 2940 catalog_manager.cc:1349] Generated new cluster ID: 54156f425fb1490ba7fcfad1a39b39fc
I20250622 13:52:29.728371 2940 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:52:29.744201 2940 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:52:29.744963 2940 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:52:29.755348 2940 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4: Generated new TSK 0
I20250622 13:52:29.755576 2940 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:52:29.760349 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:29.763190 2951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:29.763221 2950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:29.763732 2953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:29.763844 413 server_base.cc:1048] running on GCE node
I20250622 13:52:29.764254 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:29.764318 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:29.764356 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600349764353 us; error 0 us; skew 500 ppm
I20250622 13:52:29.764508 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:29.765772 413 webserver.cc:469] Webserver started at http://127.0.103.65:44929/ using document root <none> and password file <none>
I20250622 13:52:29.765977 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:29.766042 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:29.766165 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:29.766777 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "89e791d946c046aca177ba716238dd4d"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.769271 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:29.770917 2958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.771306 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:29.771407 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "89e791d946c046aca177ba716238dd4d"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.771512 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:29.788427 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:29.788941 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:29.789530 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:29.790552 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:29.790616 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.790701 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:29.790740 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.802700 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39681
I20250622 13:52:29.802721 3020 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39681 every 8 connection(s)
I20250622 13:52:29.804251 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:29.807018 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:29.807081 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:29.807751 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:29.807940 413 server_base.cc:1048] running on GCE node
I20250622 13:52:29.808228 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:29.808297 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:29.808323 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600349808320 us; error 0 us; skew 500 ppm
I20250622 13:52:29.808454 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:29.809839 413 webserver.cc:469] Webserver started at http://127.0.103.66:36427/ using document root <none> and password file <none>
I20250622 13:52:29.810088 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:29.810197 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:29.810328 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:29.810969 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "a6837d5d565346248c9717f130ab3071"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.811595 3021 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41247
I20250622 13:52:29.811720 3021 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:29.812038 3021 heartbeater.cc:507] Master 127.0.103.126:41247 requested a full tablet report, sending...
I20250622 13:52:29.812856 2894 ts_manager.cc:194] Registered new tserver with Master: 89e791d946c046aca177ba716238dd4d (127.0.103.65:39681)
I20250622 13:52:29.813719 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250622 13:52:29.813968 2894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58192
I20250622 13:52:29.815377 3033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.815768 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:52:29.815860 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "a6837d5d565346248c9717f130ab3071"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.815951 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:29.825433 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:29.825940 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:29.826388 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:29.827162 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:29.827219 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.827289 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:29.827329 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.838116 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:35329
I20250622 13:52:29.838150 3095 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:35329 every 8 connection(s)
I20250622 13:52:29.839745 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:29.842541 3099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:29.842594 3100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:29.844445 413 server_base.cc:1048] running on GCE node
W20250622 13:52:29.845379 3102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:29.845831 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:29.845902 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:29.845940 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600349845936 us; error 0 us; skew 500 ppm
I20250622 13:52:29.846071 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:29.846305 3096 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41247
I20250622 13:52:29.846416 3096 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:29.846853 3096 heartbeater.cc:507] Master 127.0.103.126:41247 requested a full tablet report, sending...
I20250622 13:52:29.847460 413 webserver.cc:469] Webserver started at http://127.0.103.67:46609/ using document root <none> and password file <none>
I20250622 13:52:29.847705 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:29.847785 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:29.847808 2894 ts_manager.cc:194] Registered new tserver with Master: a6837d5d565346248c9717f130ab3071 (127.0.103.66:35329)
I20250622 13:52:29.847890 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:29.848630 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "b40d4f5da3d74414adf8015c5cba053b"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.848802 2894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58198
I20250622 13:52:29.851279 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250622 13:52:29.852861 3107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.853255 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:29.853340 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "b40d4f5da3d74414adf8015c5cba053b"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.853477 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:29.859624 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:29.860085 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:29.860517 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:29.861244 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:29.861294 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.861346 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:29.861419 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.872069 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:40581
I20250622 13:52:29.872104 3169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:40581 every 8 connection(s)
I20250622 13:52:29.873716 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:29.876726 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:29.877816 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:29.878021 413 server_base.cc:1048] running on GCE node
W20250622 13:52:29.878705 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:29.879307 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:29.879387 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:29.879422 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600349879419 us; error 0 us; skew 500 ppm
I20250622 13:52:29.879597 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:29.880638 3170 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41247
I20250622 13:52:29.880759 3170 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:29.881055 413 webserver.cc:469] Webserver started at http://127.0.103.68:42199/ using document root <none> and password file <none>
I20250622 13:52:29.881129 3170 heartbeater.cc:507] Master 127.0.103.126:41247 requested a full tablet report, sending...
I20250622 13:52:29.881242 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:29.881296 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:29.881419 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:29.882007 2894 ts_manager.cc:194] Registered new tserver with Master: b40d4f5da3d74414adf8015c5cba053b (127.0.103.67:40581)
I20250622 13:52:29.882007 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-3-root/instance:
uuid: "853be8f0b41c49a0a597e79fb0052180"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.882717 2894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58206
I20250622 13:52:29.884425 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250622 13:52:29.885962 3181 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.886339 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:52:29.886430 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-3-root
uuid: "853be8f0b41c49a0a597e79fb0052180"
format_stamp: "Formatted at 2025-06-22 13:52:29 on dist-test-slave-tmv0"
I20250622 13:52:29.886528 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750600281110842-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:29.895089 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:29.895617 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:29.896061 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:29.896837 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:29.896889 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.896939 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:29.896977 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:29.908267 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:37153
I20250622 13:52:29.908290 3243 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:37153 every 8 connection(s)
I20250622 13:52:29.915086 3244 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41247
I20250622 13:52:29.915220 3244 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:29.915627 3244 heartbeater.cc:507] Master 127.0.103.126:41247 requested a full tablet report, sending...
I20250622 13:52:29.916519 2894 ts_manager.cc:194] Registered new tserver with Master: 853be8f0b41c49a0a597e79fb0052180 (127.0.103.68:37153)
I20250622 13:52:29.917019 413 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.007895103s
I20250622 13:52:29.917315 2894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58210
I20250622 13:52:30.815316 3021 heartbeater.cc:499] Master 127.0.103.126:41247 was elected leader, sending a full tablet report...
I20250622 13:52:30.850243 3096 heartbeater.cc:499] Master 127.0.103.126:41247 was elected leader, sending a full tablet report...
I20250622 13:52:30.884073 3170 heartbeater.cc:499] Master 127.0.103.126:41247 was elected leader, sending a full tablet report...
I20250622 13:52:30.918658 3244 heartbeater.cc:499] Master 127.0.103.126:41247 was elected leader, sending a full tablet report...
I20250622 13:52:30.952500 413 test_util.cc:276] Using random seed: 1745873967
I20250622 13:52:30.963874 2894 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58226:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 13:52:30.991019 3209 tablet_service.cc:1468] Processing CreateTablet for tablet bc69c15afd694774a9d7c43c90a205f4 (DEFAULT_TABLE table=test-workload [id=38ed82fcaf84421789ceaec3dd0db2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:52:30.991510 3209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc69c15afd694774a9d7c43c90a205f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:30.994113 3061 tablet_service.cc:1468] Processing CreateTablet for tablet bc69c15afd694774a9d7c43c90a205f4 (DEFAULT_TABLE table=test-workload [id=38ed82fcaf84421789ceaec3dd0db2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:52:30.994211 3135 tablet_service.cc:1468] Processing CreateTablet for tablet bc69c15afd694774a9d7c43c90a205f4 (DEFAULT_TABLE table=test-workload [id=38ed82fcaf84421789ceaec3dd0db2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:52:30.994490 3061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc69c15afd694774a9d7c43c90a205f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:30.994671 3135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc69c15afd694774a9d7c43c90a205f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:30.999387 3266 tablet_bootstrap.cc:492] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b: Bootstrap starting.
I20250622 13:52:31.000049 3265 tablet_bootstrap.cc:492] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180: Bootstrap starting.
I20250622 13:52:31.001802 3267 tablet_bootstrap.cc:492] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071: Bootstrap starting.
I20250622 13:52:31.001860 3266 tablet_bootstrap.cc:654] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:31.002305 3265 tablet_bootstrap.cc:654] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:31.003494 3267 tablet_bootstrap.cc:654] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:31.004122 3266 tablet_bootstrap.cc:492] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b: No bootstrap required, opened a new log
I20250622 13:52:31.004297 3265 tablet_bootstrap.cc:492] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180: No bootstrap required, opened a new log
I20250622 13:52:31.004298 3266 ts_tablet_manager.cc:1397] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250622 13:52:31.004604 3265 ts_tablet_manager.cc:1397] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250622 13:52:31.005313 3266 raft_consensus.cc:357] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.005529 3266 raft_consensus.cc:383] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:31.005612 3266 raft_consensus.cc:738] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40d4f5da3d74414adf8015c5cba053b, State: Initialized, Role: FOLLOWER
I20250622 13:52:31.005683 3265 raft_consensus.cc:357] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.005905 3265 raft_consensus.cc:383] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:31.005856 3266 consensus_queue.cc:260] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.006032 3265 raft_consensus.cc:738] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853be8f0b41c49a0a597e79fb0052180, State: Initialized, Role: FOLLOWER
I20250622 13:52:31.006268 3265 consensus_queue.cc:260] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.007190 3265 ts_tablet_manager.cc:1428] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:31.009868 3266 ts_tablet_manager.cc:1428] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20250622 13:52:31.011566 3267 tablet_bootstrap.cc:492] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071: No bootstrap required, opened a new log
I20250622 13:52:31.011705 3267 ts_tablet_manager.cc:1397] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20250622 13:52:31.012468 3267 raft_consensus.cc:357] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.012601 3267 raft_consensus.cc:383] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:31.012655 3267 raft_consensus.cc:738] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6837d5d565346248c9717f130ab3071, State: Initialized, Role: FOLLOWER
I20250622 13:52:31.012845 3267 consensus_queue.cc:260] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.013854 3267 ts_tablet_manager.cc:1428] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20250622 13:52:31.090260 3097 tablet.cc:2378] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:52:31.137391 3271 raft_consensus.cc:491] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:31.137750 3271 raft_consensus.cc:513] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.138662 3271 leader_election.cc:290] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 853be8f0b41c49a0a597e79fb0052180 (127.0.103.68:37153), a6837d5d565346248c9717f130ab3071 (127.0.103.66:35329)
I20250622 13:52:31.145113 3071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc69c15afd694774a9d7c43c90a205f4" candidate_uuid: "b40d4f5da3d74414adf8015c5cba053b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6837d5d565346248c9717f130ab3071" is_pre_election: true
I20250622 13:52:31.145116 3219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc69c15afd694774a9d7c43c90a205f4" candidate_uuid: "b40d4f5da3d74414adf8015c5cba053b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "853be8f0b41c49a0a597e79fb0052180" is_pre_election: true
I20250622 13:52:31.145408 3071 raft_consensus.cc:2466] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40d4f5da3d74414adf8015c5cba053b in term 0.
I20250622 13:52:31.145438 3219 raft_consensus.cc:2466] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40d4f5da3d74414adf8015c5cba053b in term 0.
I20250622 13:52:31.145850 3108 leader_election.cc:304] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6837d5d565346248c9717f130ab3071, b40d4f5da3d74414adf8015c5cba053b; no voters:
I20250622 13:52:31.146100 3271 raft_consensus.cc:2802] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:31.146198 3271 raft_consensus.cc:491] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:31.146276 3271 raft_consensus.cc:3058] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:31.148172 3271 raft_consensus.cc:513] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.148679 3271 leader_election.cc:290] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [CANDIDATE]: Term 1 election: Requested vote from peers 853be8f0b41c49a0a597e79fb0052180 (127.0.103.68:37153), a6837d5d565346248c9717f130ab3071 (127.0.103.66:35329)
I20250622 13:52:31.149158 3071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc69c15afd694774a9d7c43c90a205f4" candidate_uuid: "b40d4f5da3d74414adf8015c5cba053b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6837d5d565346248c9717f130ab3071"
I20250622 13:52:31.149161 3219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc69c15afd694774a9d7c43c90a205f4" candidate_uuid: "b40d4f5da3d74414adf8015c5cba053b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "853be8f0b41c49a0a597e79fb0052180"
I20250622 13:52:31.149395 3219 raft_consensus.cc:3058] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:31.149421 3071 raft_consensus.cc:3058] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:31.152118 3071 raft_consensus.cc:2466] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40d4f5da3d74414adf8015c5cba053b in term 1.
I20250622 13:52:31.152118 3219 raft_consensus.cc:2466] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40d4f5da3d74414adf8015c5cba053b in term 1.
I20250622 13:52:31.152537 3108 leader_election.cc:304] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6837d5d565346248c9717f130ab3071, b40d4f5da3d74414adf8015c5cba053b; no voters:
I20250622 13:52:31.152776 3271 raft_consensus.cc:2802] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:31.153201 3271 raft_consensus.cc:695] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 1 LEADER]: Becoming Leader. State: Replica: b40d4f5da3d74414adf8015c5cba053b, State: Running, Role: LEADER
I20250622 13:52:31.153491 3271 consensus_queue.cc:237] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } }
I20250622 13:52:31.156028 2893 catalog_manager.cc:5582] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b reported cstate change: term changed from 0 to 1, leader changed from <none> to b40d4f5da3d74414adf8015c5cba053b (127.0.103.67). New cstate: current_term: 1 leader_uuid: "b40d4f5da3d74414adf8015c5cba053b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40d4f5da3d74414adf8015c5cba053b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40581 } health_report { overall_health: HEALTHY } } }
I20250622 13:52:31.619532 3271 consensus_queue.cc:1035] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [LEADER]: Connected to new peer: Peer: permanent_uuid: "853be8f0b41c49a0a597e79fb0052180" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:31.624747 3281 consensus_queue.cc:1035] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6837d5d565346248c9717f130ab3071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:31.727309 2947 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:31.727800 2947 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:31.728020 2947 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:33.728514 2947 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:33.729058 2947 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:33.729187 2947 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:35.729604 2947 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:35.730052 2947 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:35.730159 2947 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:37.730573 2947 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:37.731029 2947 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:37.731145 2947 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:39.731684 2947 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:39.732184 2947 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:39.732285 2947 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:41.732728 2947 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:41.733233 2947 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:41.733384 2947 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:42.218629 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:52:42.240147 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:42.253082 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:52:42.256503 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:52:42.265456 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:42.265781 413 tablet_replica.cc:331] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071: stopping tablet replica
I20250622 13:52:42.265941 413 raft_consensus.cc:2241] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:52:42.266129 413 raft_consensus.cc:2270] T bc69c15afd694774a9d7c43c90a205f4 P a6837d5d565346248c9717f130ab3071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:42.280026 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:52:42.282864 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:52:42.291385 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:42.291636 413 tablet_replica.cc:331] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b: stopping tablet replica
I20250622 13:52:42.291818 413 raft_consensus.cc:2241] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:42.292104 413 raft_consensus.cc:2270] T bc69c15afd694774a9d7c43c90a205f4 P b40d4f5da3d74414adf8015c5cba053b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:42.306038 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:52:42.308987 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250622 13:52:42.316970 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:42.317250 413 tablet_replica.cc:331] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180: stopping tablet replica
I20250622 13:52:42.317447 413 raft_consensus.cc:2241] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:52:42.317629 413 raft_consensus.cc:2270] T bc69c15afd694774a9d7c43c90a205f4 P 853be8f0b41c49a0a597e79fb0052180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:42.331805 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250622 13:52:42.334993 413 master.cc:561] Master@127.0.103.126:41247 shutting down...
I20250622 13:52:42.342242 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:42.342494 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:42.342594 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 1b6fd06a3d504d75a029c6a67cf4c2a4: stopping tablet replica
I20250622 13:52:42.356097 413 master.cc:583] Master@127.0.103.126:41247 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12696 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20250622 13:52:42.368307 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:38351
I20250622 13:52:42.368811 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:42.371538 3289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:42.371564 3288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:42.373126 3291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:42.373149 413 server_base.cc:1048] running on GCE node
I20250622 13:52:42.373514 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:42.373577 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:42.373615 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600362373610 us; error 0 us; skew 500 ppm
I20250622 13:52:42.373769 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:42.378832 413 webserver.cc:469] Webserver started at http://127.0.103.126:42387/ using document root <none> and password file <none>
I20250622 13:52:42.379132 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:42.379204 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:42.379343 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:42.380067 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "a189d1f247284d6cb4965e6b223144f7"
format_stamp: "Formatted at 2025-06-22 13:52:42 on dist-test-slave-tmv0"
I20250622 13:52:42.382748 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250622 13:52:42.384425 3296 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.384771 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:42.384855 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "a189d1f247284d6cb4965e6b223144f7"
format_stamp: "Formatted at 2025-06-22 13:52:42 on dist-test-slave-tmv0"
I20250622 13:52:42.384960 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:42.393563 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:42.394042 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:42.406119 3347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:38351 every 8 connection(s)
I20250622 13:52:42.406121 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:38351
I20250622 13:52:42.407702 3348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:42.411818 3348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7: Bootstrap starting.
I20250622 13:52:42.413015 3348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:42.414549 3348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7: No bootstrap required, opened a new log
I20250622 13:52:42.415128 3348 raft_consensus.cc:357] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a189d1f247284d6cb4965e6b223144f7" member_type: VOTER }
I20250622 13:52:42.415254 3348 raft_consensus.cc:383] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:42.415323 3348 raft_consensus.cc:738] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a189d1f247284d6cb4965e6b223144f7, State: Initialized, Role: FOLLOWER
I20250622 13:52:42.415520 3348 consensus_queue.cc:260] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a189d1f247284d6cb4965e6b223144f7" member_type: VOTER }
I20250622 13:52:42.415633 3348 raft_consensus.cc:397] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:52:42.415699 3348 raft_consensus.cc:491] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:52:42.415781 3348 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:42.417685 3348 raft_consensus.cc:513] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a189d1f247284d6cb4965e6b223144f7" member_type: VOTER }
I20250622 13:52:42.417910 3348 leader_election.cc:304] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a189d1f247284d6cb4965e6b223144f7; no voters:
I20250622 13:52:42.418299 3348 leader_election.cc:290] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:52:42.418435 3351 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:42.418879 3351 raft_consensus.cc:695] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 1 LEADER]: Becoming Leader. State: Replica: a189d1f247284d6cb4965e6b223144f7, State: Running, Role: LEADER
I20250622 13:52:42.419102 3348 sys_catalog.cc:564] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:52:42.419134 3351 consensus_queue.cc:237] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a189d1f247284d6cb4965e6b223144f7" member_type: VOTER }
I20250622 13:52:42.420256 3353 sys_catalog.cc:455] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a189d1f247284d6cb4965e6b223144f7. Latest consensus state: current_term: 1 leader_uuid: "a189d1f247284d6cb4965e6b223144f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a189d1f247284d6cb4965e6b223144f7" member_type: VOTER } }
I20250622 13:52:42.420434 3353 sys_catalog.cc:458] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:42.420207 3352 sys_catalog.cc:455] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a189d1f247284d6cb4965e6b223144f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a189d1f247284d6cb4965e6b223144f7" member_type: VOTER } }
I20250622 13:52:42.420650 3352 sys_catalog.cc:458] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:42.421876 3356 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:52:42.423504 3356 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:52:42.425770 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:52:42.427219 3356 catalog_manager.cc:1349] Generated new cluster ID: 768674bad8324ce2bd74893b5362b17e
I20250622 13:52:42.427300 3356 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:52:42.434659 3356 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:52:42.435415 3356 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:52:42.444263 3356 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7: Generated new TSK 0
I20250622 13:52:42.444519 3356 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:52:42.458936 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:42.461575 3369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:42.462002 3372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:42.462380 3370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:42.462392 413 server_base.cc:1048] running on GCE node
I20250622 13:52:42.462733 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:42.462790 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:42.462813 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600362462809 us; error 0 us; skew 500 ppm
I20250622 13:52:42.462967 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:42.464126 413 webserver.cc:469] Webserver started at http://127.0.103.65:37133/ using document root <none> and password file <none>
I20250622 13:52:42.464349 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:42.464411 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:42.464532 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:42.465167 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "4614ff8e120d42c6864ce82a9d70fea0"
format_stamp: "Formatted at 2025-06-22 13:52:42 on dist-test-slave-tmv0"
I20250622 13:52:42.467615 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:42.469166 3377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.469549 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:42.469647 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "4614ff8e120d42c6864ce82a9d70fea0"
format_stamp: "Formatted at 2025-06-22 13:52:42 on dist-test-slave-tmv0"
I20250622 13:52:42.469753 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:42.495258 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:42.495858 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:42.496429 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:42.497246 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:42.497299 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.497402 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:42.497447 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.513836 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:33475
I20250622 13:52:42.513851 3439 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:33475 every 8 connection(s)
I20250622 13:52:42.515434 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:42.517954 3444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:42.518141 3445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:42.522588 3440 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38351
I20250622 13:52:42.522691 3440 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:42.522974 413 server_base.cc:1048] running on GCE node
W20250622 13:52:42.522887 3447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:42.523049 3440 heartbeater.cc:507] Master 127.0.103.126:38351 requested a full tablet report, sending...
I20250622 13:52:42.523373 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:42.523447 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:42.523479 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600362523475 us; error 0 us; skew 500 ppm
I20250622 13:52:42.523648 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:42.524016 3313 ts_manager.cc:194] Registered new tserver with Master: 4614ff8e120d42c6864ce82a9d70fea0 (127.0.103.65:33475)
I20250622 13:52:42.524904 413 webserver.cc:469] Webserver started at http://127.0.103.66:38969/ using document root <none> and password file <none>
I20250622 13:52:42.525097 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:42.525164 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:42.525278 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:42.525249 3313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60572
I20250622 13:52:42.525954 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "b4dc767c4add47268a5ffe5bf25beeb8"
format_stamp: "Formatted at 2025-06-22 13:52:42 on dist-test-slave-tmv0"
I20250622 13:52:42.528107 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250622 13:52:42.529398 3452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.529688 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:42.529778 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "b4dc767c4add47268a5ffe5bf25beeb8"
format_stamp: "Formatted at 2025-06-22 13:52:42 on dist-test-slave-tmv0"
I20250622 13:52:42.529875 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:42.541496 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:42.541970 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:42.542490 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:42.543262 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:42.543334 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.543417 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:42.543459 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.554922 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:40967
I20250622 13:52:42.554942 3514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:40967 every 8 connection(s)
I20250622 13:52:42.556416 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:42.559098 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:42.560269 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:42.561928 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:42.561965 413 server_base.cc:1048] running on GCE node
I20250622 13:52:42.562424 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:42.562500 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:42.562534 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600362562530 us; error 0 us; skew 500 ppm
I20250622 13:52:42.562722 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:42.563261 3515 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38351
I20250622 13:52:42.563386 3515 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:42.563776 3515 heartbeater.cc:507] Master 127.0.103.126:38351 requested a full tablet report, sending...
I20250622 13:52:42.564112 413 webserver.cc:469] Webserver started at http://127.0.103.67:44885/ using document root <none> and password file <none>
I20250622 13:52:42.564350 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:42.564430 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:42.564546 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:42.564713 3313 ts_manager.cc:194] Registered new tserver with Master: b4dc767c4add47268a5ffe5bf25beeb8 (127.0.103.66:40967)
I20250622 13:52:42.565240 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "0363a64caf0343b9814a956a3e6d5175"
format_stamp: "Formatted at 2025-06-22 13:52:42 on dist-test-slave-tmv0"
I20250622 13:52:42.565696 3313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60582
I20250622 13:52:42.567960 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:42.569293 3526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.569623 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:42.569705 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "0363a64caf0343b9814a956a3e6d5175"
format_stamp: "Formatted at 2025-06-22 13:52:42 on dist-test-slave-tmv0"
I20250622 13:52:42.569805 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:42.580576 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:42.581123 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:42.581669 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:42.582430 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:42.582480 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.582545 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:42.582587 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:42.593911 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44619
I20250622 13:52:42.593946 3588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44619 every 8 connection(s)
I20250622 13:52:42.600049 3589 heartbeater.cc:344] Connected to a master server at 127.0.103.126:38351
I20250622 13:52:42.600174 3589 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:42.600517 3589 heartbeater.cc:507] Master 127.0.103.126:38351 requested a full tablet report, sending...
I20250622 13:52:42.601269 3313 ts_manager.cc:194] Registered new tserver with Master: 0363a64caf0343b9814a956a3e6d5175 (127.0.103.67:44619)
I20250622 13:52:42.601441 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006699061s
I20250622 13:52:42.602010 3313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60590
I20250622 13:52:43.526536 3440 heartbeater.cc:499] Master 127.0.103.126:38351 was elected leader, sending a full tablet report...
I20250622 13:52:43.567157 3515 heartbeater.cc:499] Master 127.0.103.126:38351 was elected leader, sending a full tablet report...
I20250622 13:52:43.603127 3589 heartbeater.cc:499] Master 127.0.103.126:38351 was elected leader, sending a full tablet report...
I20250622 13:52:43.635660 413 test_util.cc:276] Using random seed: 1758557127
I20250622 13:52:43.646448 3313 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60614:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:52:43.647042 3313 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:52:43.672926 3480 tablet_service.cc:1468] Processing CreateTablet for tablet 0b515a9f388d40338d5a2021c103a778 (DEFAULT_TABLE table=test-workload [id=c72c13aae4044e1884ace6459daf5adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:52:43.673499 3480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b515a9f388d40338d5a2021c103a778. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:43.673458 3405 tablet_service.cc:1468] Processing CreateTablet for tablet 0b515a9f388d40338d5a2021c103a778 (DEFAULT_TABLE table=test-workload [id=c72c13aae4044e1884ace6459daf5adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:52:43.673792 3405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b515a9f388d40338d5a2021c103a778. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:43.674106 3554 tablet_service.cc:1468] Processing CreateTablet for tablet 0b515a9f388d40338d5a2021c103a778 (DEFAULT_TABLE table=test-workload [id=c72c13aae4044e1884ace6459daf5adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:52:43.674480 3554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b515a9f388d40338d5a2021c103a778. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:43.678900 3610 tablet_bootstrap.cc:492] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175: Bootstrap starting.
I20250622 13:52:43.680208 3610 tablet_bootstrap.cc:654] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:43.680769 3611 tablet_bootstrap.cc:492] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8: Bootstrap starting.
I20250622 13:52:43.681222 3612 tablet_bootstrap.cc:492] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0: Bootstrap starting.
I20250622 13:52:43.682396 3610 tablet_bootstrap.cc:492] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175: No bootstrap required, opened a new log
I20250622 13:52:43.682533 3610 ts_tablet_manager.cc:1397] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250622 13:52:43.682657 3611 tablet_bootstrap.cc:654] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:43.683145 3610 raft_consensus.cc:357] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.683282 3610 raft_consensus.cc:383] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:43.683359 3610 raft_consensus.cc:738] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0363a64caf0343b9814a956a3e6d5175, State: Initialized, Role: FOLLOWER
I20250622 13:52:43.683586 3610 consensus_queue.cc:260] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.683665 3612 tablet_bootstrap.cc:654] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:43.684615 3610 ts_tablet_manager.cc:1428] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:43.684841 3611 tablet_bootstrap.cc:492] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8: No bootstrap required, opened a new log
I20250622 13:52:43.684973 3611 ts_tablet_manager.cc:1397] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:52:43.685647 3611 raft_consensus.cc:357] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.685807 3611 raft_consensus.cc:383] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:43.685928 3611 raft_consensus.cc:738] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4dc767c4add47268a5ffe5bf25beeb8, State: Initialized, Role: FOLLOWER
I20250622 13:52:43.686187 3611 consensus_queue.cc:260] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.687316 3611 ts_tablet_manager.cc:1428] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:43.687930 3612 tablet_bootstrap.cc:492] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0: No bootstrap required, opened a new log
I20250622 13:52:43.688059 3612 ts_tablet_manager.cc:1397] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250622 13:52:43.688697 3612 raft_consensus.cc:357] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.688858 3612 raft_consensus.cc:383] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:43.688898 3612 raft_consensus.cc:738] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4614ff8e120d42c6864ce82a9d70fea0, State: Initialized, Role: FOLLOWER
I20250622 13:52:43.689077 3612 consensus_queue.cc:260] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.689766 3612 ts_tablet_manager.cc:1428] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:52:43.802628 3617 raft_consensus.cc:491] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:43.802829 3617 raft_consensus.cc:513] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.803653 3617 leader_election.cc:290] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4614ff8e120d42c6864ce82a9d70fea0 (127.0.103.65:33475), 0363a64caf0343b9814a956a3e6d5175 (127.0.103.67:44619)
I20250622 13:52:43.809152 3415 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b515a9f388d40338d5a2021c103a778" candidate_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4614ff8e120d42c6864ce82a9d70fea0" is_pre_election: true
I20250622 13:52:43.809197 3564 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b515a9f388d40338d5a2021c103a778" candidate_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0363a64caf0343b9814a956a3e6d5175" is_pre_election: true
I20250622 13:52:43.809407 3415 raft_consensus.cc:2466] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4dc767c4add47268a5ffe5bf25beeb8 in term 0.
I20250622 13:52:43.809429 3564 raft_consensus.cc:2466] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4dc767c4add47268a5ffe5bf25beeb8 in term 0.
I20250622 13:52:43.809826 3455 leader_election.cc:304] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0363a64caf0343b9814a956a3e6d5175, b4dc767c4add47268a5ffe5bf25beeb8; no voters:
I20250622 13:52:43.810043 3617 raft_consensus.cc:2802] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:43.810119 3617 raft_consensus.cc:491] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:43.810194 3617 raft_consensus.cc:3058] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:43.811856 3617 raft_consensus.cc:513] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.812311 3617 leader_election.cc:290] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [CANDIDATE]: Term 1 election: Requested vote from peers 4614ff8e120d42c6864ce82a9d70fea0 (127.0.103.65:33475), 0363a64caf0343b9814a956a3e6d5175 (127.0.103.67:44619)
I20250622 13:52:43.812716 3564 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b515a9f388d40338d5a2021c103a778" candidate_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0363a64caf0343b9814a956a3e6d5175"
I20250622 13:52:43.812716 3415 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b515a9f388d40338d5a2021c103a778" candidate_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4614ff8e120d42c6864ce82a9d70fea0"
I20250622 13:52:43.812861 3564 raft_consensus.cc:3058] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:43.812875 3415 raft_consensus.cc:3058] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:43.814649 3415 raft_consensus.cc:2466] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4dc767c4add47268a5ffe5bf25beeb8 in term 1.
I20250622 13:52:43.814651 3564 raft_consensus.cc:2466] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4dc767c4add47268a5ffe5bf25beeb8 in term 1.
I20250622 13:52:43.815023 3455 leader_election.cc:304] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0363a64caf0343b9814a956a3e6d5175, 4614ff8e120d42c6864ce82a9d70fea0, b4dc767c4add47268a5ffe5bf25beeb8; no voters:
I20250622 13:52:43.815279 3617 raft_consensus.cc:2802] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:43.815642 3617 raft_consensus.cc:695] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 1 LEADER]: Becoming Leader. State: Replica: b4dc767c4add47268a5ffe5bf25beeb8, State: Running, Role: LEADER
I20250622 13:52:43.815853 3617 consensus_queue.cc:237] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } }
I20250622 13:52:43.818089 3312 catalog_manager.cc:5582] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4dc767c4add47268a5ffe5bf25beeb8 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4dc767c4add47268a5ffe5bf25beeb8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:44.261291 3617 consensus_queue.cc:1035] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4614ff8e120d42c6864ce82a9d70fea0" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:44.265990 3621 consensus_queue.cc:1035] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0363a64caf0343b9814a956a3e6d5175" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:44.424574 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:44.425745 3366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:44.426138 3366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:44.426221 3366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:45.427258 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:46.426610 3366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:46.427090 3366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:46.427186 3366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:46.429522 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:47.431917 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:48.427595 3366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:48.428062 3366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:48.428170 3366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:48.434242 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:49.436568 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:50.428606 3366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:50.429047 3366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:50.429152 3366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:50.438894 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:51.441231 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:52.429627 3366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:52.430055 3366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:52.430142 3366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:52.443636 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:53.446049 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:54.430528 3366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:54.430946 3366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:52:54.431028 3366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:54.448464 3361 placement_policy_util.cc:407] tablet 0b515a9f388d40338d5a2021c103a778: detected majority of replicas (2 of 3) at location L0
I20250622 13:52:54.876554 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:52:54.893641 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:54.893919 413 tablet_replica.cc:331] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0: stopping tablet replica
I20250622 13:52:54.894104 413 raft_consensus.cc:2241] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:52:54.894311 413 raft_consensus.cc:2270] T 0b515a9f388d40338d5a2021c103a778 P 4614ff8e120d42c6864ce82a9d70fea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:54.908134 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:52:54.910974 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:52:54.918603 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:54.918924 413 tablet_replica.cc:331] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8: stopping tablet replica
I20250622 13:52:54.919085 413 raft_consensus.cc:2241] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:54.919399 413 raft_consensus.cc:2270] T 0b515a9f388d40338d5a2021c103a778 P b4dc767c4add47268a5ffe5bf25beeb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:54.933243 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:52:54.936048 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:52:54.944048 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:52:54.944339 413 tablet_replica.cc:331] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175: stopping tablet replica
I20250622 13:52:54.944506 413 raft_consensus.cc:2241] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:52:54.944686 413 raft_consensus.cc:2270] T 0b515a9f388d40338d5a2021c103a778 P 0363a64caf0343b9814a956a3e6d5175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:54.958336 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:52:54.961265 413 master.cc:561] Master@127.0.103.126:38351 shutting down...
I20250622 13:52:54.976189 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:52:54.976469 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:52:54.976577 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P a189d1f247284d6cb4965e6b223144f7: stopping tablet replica
I20250622 13:52:54.990180 413 master.cc:583] Master@127.0.103.126:38351 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12631 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20250622 13:52:55.000370 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:37277
I20250622 13:52:55.000880 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:55.003284 3632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:55.003422 3635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:55.003382 3633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:55.003762 413 server_base.cc:1048] running on GCE node
I20250622 13:52:55.003994 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:55.004045 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:55.004066 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600375004064 us; error 0 us; skew 500 ppm
I20250622 13:52:55.004220 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:55.005424 413 webserver.cc:469] Webserver started at http://127.0.103.126:40393/ using document root <none> and password file <none>
I20250622 13:52:55.005600 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:55.005647 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:55.005735 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:55.006335 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "ff80a419fea7490e880cbf4be482cb37"
format_stamp: "Formatted at 2025-06-22 13:52:55 on dist-test-slave-tmv0"
I20250622 13:52:55.008620 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:55.010078 3640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.010460 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:55.010550 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "ff80a419fea7490e880cbf4be482cb37"
format_stamp: "Formatted at 2025-06-22 13:52:55 on dist-test-slave-tmv0"
I20250622 13:52:55.010656 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:55.023365 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:55.023854 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:55.033851 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:37277
I20250622 13:52:55.033881 3691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:37277 every 8 connection(s)
I20250622 13:52:55.035209 3692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:55.039182 3692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37: Bootstrap starting.
I20250622 13:52:55.040354 3692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:55.041888 3692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37: No bootstrap required, opened a new log
I20250622 13:52:55.042467 3692 raft_consensus.cc:357] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a419fea7490e880cbf4be482cb37" member_type: VOTER }
I20250622 13:52:55.042583 3692 raft_consensus.cc:383] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:55.042639 3692 raft_consensus.cc:738] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff80a419fea7490e880cbf4be482cb37, State: Initialized, Role: FOLLOWER
I20250622 13:52:55.042860 3692 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a419fea7490e880cbf4be482cb37" member_type: VOTER }
I20250622 13:52:55.042963 3692 raft_consensus.cc:397] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:52:55.043018 3692 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:52:55.043102 3692 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:55.044764 3692 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a419fea7490e880cbf4be482cb37" member_type: VOTER }
I20250622 13:52:55.044945 3692 leader_election.cc:304] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff80a419fea7490e880cbf4be482cb37; no voters:
I20250622 13:52:55.045387 3692 leader_election.cc:290] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:52:55.045526 3695 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:55.045980 3695 raft_consensus.cc:695] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 1 LEADER]: Becoming Leader. State: Replica: ff80a419fea7490e880cbf4be482cb37, State: Running, Role: LEADER
I20250622 13:52:55.046187 3695 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a419fea7490e880cbf4be482cb37" member_type: VOTER }
I20250622 13:52:55.046257 3692 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:52:55.047019 3696 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff80a419fea7490e880cbf4be482cb37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a419fea7490e880cbf4be482cb37" member_type: VOTER } }
I20250622 13:52:55.047086 3697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff80a419fea7490e880cbf4be482cb37. Latest consensus state: current_term: 1 leader_uuid: "ff80a419fea7490e880cbf4be482cb37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a419fea7490e880cbf4be482cb37" member_type: VOTER } }
I20250622 13:52:55.047286 3696 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:55.047320 3697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [sys.catalog]: This master's current role is: LEADER
I20250622 13:52:55.048384 3700 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:52:55.049732 3700 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:52:55.051270 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:52:55.052410 3700 catalog_manager.cc:1349] Generated new cluster ID: ea2c40f5a51f46baabbae54b57c8b80a
I20250622 13:52:55.052491 3700 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:52:55.059605 3700 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:52:55.060400 3700 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:52:55.068238 3700 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37: Generated new TSK 0
I20250622 13:52:55.068473 3700 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:52:55.084005 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:55.086687 3713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:55.086769 3714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:55.087288 3716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:55.087606 413 server_base.cc:1048] running on GCE node
I20250622 13:52:55.087821 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:55.087878 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:55.087906 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600375087903 us; error 0 us; skew 500 ppm
I20250622 13:52:55.088017 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:55.089164 413 webserver.cc:469] Webserver started at http://127.0.103.65:38783/ using document root <none> and password file <none>
I20250622 13:52:55.089350 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:55.089448 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:55.089546 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:55.090140 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
format_stamp: "Formatted at 2025-06-22 13:52:55 on dist-test-slave-tmv0"
I20250622 13:52:55.092473 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250622 13:52:55.093992 3721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.094324 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:55.094419 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
format_stamp: "Formatted at 2025-06-22 13:52:55 on dist-test-slave-tmv0"
I20250622 13:52:55.094517 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:55.108397 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:55.108927 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:55.109513 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:55.110368 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:55.110424 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.110495 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:55.110529 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.122148 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:40517
I20250622 13:52:55.122169 3783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:40517 every 8 connection(s)
I20250622 13:52:55.123694 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:55.126600 3789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:55.126751 3788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:55.127066 3791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:55.127437 413 server_base.cc:1048] running on GCE node
I20250622 13:52:55.127698 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:55.127760 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:55.127799 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600375127794 us; error 0 us; skew 500 ppm
I20250622 13:52:55.127938 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:55.129314 413 webserver.cc:469] Webserver started at http://127.0.103.66:40361/ using document root <none> and password file <none>
I20250622 13:52:55.129547 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:55.129624 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:55.129745 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:55.130466 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
format_stamp: "Formatted at 2025-06-22 13:52:55 on dist-test-slave-tmv0"
I20250622 13:52:55.130637 3784 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37277
I20250622 13:52:55.130733 3784 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:55.130995 3784 heartbeater.cc:507] Master 127.0.103.126:37277 requested a full tablet report, sending...
I20250622 13:52:55.131788 3657 ts_manager.cc:194] Registered new tserver with Master: d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517)
I20250622 13:52:55.133162 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58370
I20250622 13:52:55.133270 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250622 13:52:55.134861 3796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.135226 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:52:55.135308 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
format_stamp: "Formatted at 2025-06-22 13:52:55 on dist-test-slave-tmv0"
I20250622 13:52:55.135388 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:55.162379 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:55.162889 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:55.163394 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:55.164157 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:55.164211 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.164259 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:55.164287 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.175699 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:34391
I20250622 13:52:55.175719 3858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:34391 every 8 connection(s)
I20250622 13:52:55.177243 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:55.179847 3862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:55.180763 3865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:55.180879 3863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:55.182554 413 server_base.cc:1048] running on GCE node
I20250622 13:52:55.182825 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:55.182876 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:55.182899 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600375182897 us; error 0 us; skew 500 ppm
I20250622 13:52:55.182991 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:55.183847 3859 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37277
I20250622 13:52:55.183979 3859 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:55.184316 3859 heartbeater.cc:507] Master 127.0.103.126:37277 requested a full tablet report, sending...
I20250622 13:52:55.184481 413 webserver.cc:469] Webserver started at http://127.0.103.67:36211/ using document root <none> and password file <none>
I20250622 13:52:55.184693 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:55.184762 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:55.184878 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:55.185174 3657 ts_manager.cc:194] Registered new tserver with Master: 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:55.185565 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "3377fd0584c546c1a938e3a1204b2be3"
format_stamp: "Formatted at 2025-06-22 13:52:55 on dist-test-slave-tmv0"
I20250622 13:52:55.185945 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58380
I20250622 13:52:55.188062 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:55.189605 3870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.189932 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:52:55.190024 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "3377fd0584c546c1a938e3a1204b2be3"
format_stamp: "Formatted at 2025-06-22 13:52:55 on dist-test-slave-tmv0"
I20250622 13:52:55.190127 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:55.202065 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:55.202589 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:55.203155 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:55.203969 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:55.204025 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.204102 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:55.204138 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:55.215629 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:33649
I20250622 13:52:55.215667 3932 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:33649 every 8 connection(s)
I20250622 13:52:55.222554 3933 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37277
I20250622 13:52:55.222692 3933 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:55.223024 3933 heartbeater.cc:507] Master 127.0.103.126:37277 requested a full tablet report, sending...
I20250622 13:52:55.223802 3657 ts_manager.cc:194] Registered new tserver with Master: 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:55.224524 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58396
I20250622 13:52:55.224632 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008083482s
I20250622 13:52:56.134465 3784 heartbeater.cc:499] Master 127.0.103.126:37277 was elected leader, sending a full tablet report...
I20250622 13:52:56.187179 3859 heartbeater.cc:499] Master 127.0.103.126:37277 was elected leader, sending a full tablet report...
I20250622 13:52:56.226064 3933 heartbeater.cc:499] Master 127.0.103.126:37277 was elected leader, sending a full tablet report...
I20250622 13:52:56.252991 413 test_util.cc:276] Using random seed: 1771174433
I20250622 13:52:56.268208 3657 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58400:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:52:56.269007 3657 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:52:56.296788 3749 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa8fa3d766949b99d0538f8b76498d5 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250622 13:52:56.296968 3824 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa8fa3d766949b99d0538f8b76498d5 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250622 13:52:56.297080 3745 tablet_service.cc:1468] Processing CreateTablet for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250622 13:52:56.297389 3749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa8fa3d766949b99d0538f8b76498d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.297469 3824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa8fa3d766949b99d0538f8b76498d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.297688 3823 tablet_service.cc:1468] Processing CreateTablet for tablet 91736fef6df543c4953497186dfe1c5e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250622 13:52:56.298040 3823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91736fef6df543c4953497186dfe1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.298087 3748 tablet_service.cc:1468] Processing CreateTablet for tablet 91736fef6df543c4953497186dfe1c5e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250622 13:52:56.298436 3748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91736fef6df543c4953497186dfe1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.301712 3747 tablet_service.cc:1468] Processing CreateTablet for tablet 677dddfcdf864e7f9b997d196b58bc78 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250622 13:52:56.302093 3747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 677dddfcdf864e7f9b997d196b58bc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.304129 3741 tablet_service.cc:1468] Processing CreateTablet for tablet d317ca8305314d45b47fa1014f3b5162 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250622 13:52:56.304543 3741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d317ca8305314d45b47fa1014f3b5162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.305691 3897 tablet_service.cc:1468] Processing CreateTablet for tablet 91736fef6df543c4953497186dfe1c5e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250622 13:52:56.305819 3746 tablet_service.cc:1468] Processing CreateTablet for tablet b045aeb5889a4b88b8ee72370f2a7e39 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250622 13:52:56.306093 3897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91736fef6df543c4953497186dfe1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.306346 3746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b045aeb5889a4b88b8ee72370f2a7e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.309638 3745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.309746 3896 tablet_service.cc:1468] Processing CreateTablet for tablet 677dddfcdf864e7f9b997d196b58bc78 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250622 13:52:56.309847 3744 tablet_service.cc:1468] Processing CreateTablet for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250622 13:52:56.310138 3896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 677dddfcdf864e7f9b997d196b58bc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.310364 3744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.313751 3895 tablet_service.cc:1468] Processing CreateTablet for tablet b045aeb5889a4b88b8ee72370f2a7e39 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250622 13:52:56.313830 3743 tablet_service.cc:1468] Processing CreateTablet for tablet bac2340a96e94c219f59c12945eb8bf6 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250622 13:52:56.314183 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b045aeb5889a4b88b8ee72370f2a7e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.314380 3743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac2340a96e94c219f59c12945eb8bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.318272 3742 tablet_service.cc:1468] Processing CreateTablet for tablet f10daa2b314a4660bd36fa7902789a8e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250622 13:52:56.320320 3822 tablet_service.cc:1468] Processing CreateTablet for tablet 677dddfcdf864e7f9b997d196b58bc78 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250622 13:52:56.320914 3822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 677dddfcdf864e7f9b997d196b58bc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.321898 3742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10daa2b314a4660bd36fa7902789a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.324559 3821 tablet_service.cc:1468] Processing CreateTablet for tablet b045aeb5889a4b88b8ee72370f2a7e39 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250622 13:52:56.324756 3819 tablet_service.cc:1468] Processing CreateTablet for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250622 13:52:56.325232 3821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b045aeb5889a4b88b8ee72370f2a7e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.333148 3894 tablet_service.cc:1468] Processing CreateTablet for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250622 13:52:56.333838 3894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.336099 3745 tablet_service.cc:1468] Processing CreateTablet for tablet 6fd3fa41ece64410b7448f59d5db9ba7 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250622 13:52:56.336692 3745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd3fa41ece64410b7448f59d5db9ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.337095 3893 tablet_service.cc:1468] Processing CreateTablet for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250622 13:52:56.337466 3898 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa8fa3d766949b99d0538f8b76498d5 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250622 13:52:56.336503 3820 tablet_service.cc:1468] Processing CreateTablet for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250622 13:52:56.337710 3741 tablet_service.cc:1468] Processing CreateTablet for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250622 13:52:56.338614 3819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.339874 3824 tablet_service.cc:1468] Processing CreateTablet for tablet 6fd3fa41ece64410b7448f59d5db9ba7 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250622 13:52:56.340704 3892 tablet_service.cc:1468] Processing CreateTablet for tablet bac2340a96e94c219f59c12945eb8bf6 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250622 13:52:56.341564 3816 tablet_service.cc:1468] Processing CreateTablet for tablet d317ca8305314d45b47fa1014f3b5162 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250622 13:52:56.341624 3891 tablet_service.cc:1468] Processing CreateTablet for tablet f10daa2b314a4660bd36fa7902789a8e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250622 13:52:56.344120 3889 tablet_service.cc:1468] Processing CreateTablet for tablet c9260a075f104769bc81ca7eb0c76cb2 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250622 13:52:56.347762 3893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.348426 3741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5246e7fae8d44ba6a5e3ac826f9988f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.349586 3818 tablet_service.cc:1468] Processing CreateTablet for tablet bac2340a96e94c219f59c12945eb8bf6 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250622 13:52:56.350186 3820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.354071 3824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd3fa41ece64410b7448f59d5db9ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.352188 3890 tablet_service.cc:1468] Processing CreateTablet for tablet d317ca8305314d45b47fa1014f3b5162 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250622 13:52:56.354722 3892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac2340a96e94c219f59c12945eb8bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.356151 3817 tablet_service.cc:1468] Processing CreateTablet for tablet f10daa2b314a4660bd36fa7902789a8e (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250622 13:52:56.356752 3740 tablet_service.cc:1468] Processing CreateTablet for tablet c9260a075f104769bc81ca7eb0c76cb2 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250622 13:52:56.357205 3740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9260a075f104769bc81ca7eb0c76cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.362859 3891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10daa2b314a4660bd36fa7902789a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.362986 3816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d317ca8305314d45b47fa1014f3b5162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.363996 3818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac2340a96e94c219f59c12945eb8bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.365496 3817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10daa2b314a4660bd36fa7902789a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.365700 3815 tablet_service.cc:1468] Processing CreateTablet for tablet c9260a075f104769bc81ca7eb0c76cb2 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250622 13:52:56.367512 3898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa8fa3d766949b99d0538f8b76498d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.367980 3889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9260a075f104769bc81ca7eb0c76cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.371085 3897 tablet_service.cc:1468] Processing CreateTablet for tablet 6fd3fa41ece64410b7448f59d5db9ba7 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250622 13:52:56.371654 3897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd3fa41ece64410b7448f59d5db9ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.373631 3890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d317ca8305314d45b47fa1014f3b5162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.379624 3896 tablet_service.cc:1468] Processing CreateTablet for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250622 13:52:56.361451 3823 tablet_service.cc:1468] Processing CreateTablet for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 (DEFAULT_TABLE table=test-workload [id=87e5125c90424c10bc2da4f561da8175]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250622 13:52:56.380281 3896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5246e7fae8d44ba6a5e3ac826f9988f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.385989 3815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9260a075f104769bc81ca7eb0c76cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.397905 3823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5246e7fae8d44ba6a5e3ac826f9988f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:56.398336 3953 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.402225 3954 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.410152 3955 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.422286 3953 tablet_bootstrap.cc:654] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.424096 3954 tablet_bootstrap.cc:654] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.425243 3955 tablet_bootstrap.cc:654] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.426599 3954 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.426795 3954 ts_tablet_manager.cc:1397] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20250622 13:52:56.427791 3954 raft_consensus.cc:357] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.428081 3954 raft_consensus.cc:383] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.428187 3954 raft_consensus.cc:738] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.428496 3954 consensus_queue.cc:260] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.429673 3954 ts_tablet_manager.cc:1428] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250622 13:52:56.430101 3954 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.432514 3954 tablet_bootstrap.cc:654] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.434803 3953 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.434973 3953 ts_tablet_manager.cc:1397] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.009s
I20250622 13:52:56.435489 3955 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.435616 3953 raft_consensus.cc:357] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.435695 3955 ts_tablet_manager.cc:1397] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.003s
I20250622 13:52:56.435799 3953 raft_consensus.cc:383] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.435948 3953 raft_consensus.cc:738] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.436230 3953 consensus_queue.cc:260] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.436579 3955 raft_consensus.cc:357] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.436735 3955 raft_consensus.cc:383] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.436796 3955 raft_consensus.cc:738] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.437040 3955 consensus_queue.cc:260] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.437827 3953 ts_tablet_manager.cc:1428] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250622 13:52:56.438176 3953 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.440193 3954 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.440271 3953 tablet_bootstrap.cc:654] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.440380 3954 ts_tablet_manager.cc:1397] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20250622 13:52:56.441427 3954 raft_consensus.cc:357] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.441676 3954 raft_consensus.cc:383] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.441768 3954 raft_consensus.cc:738] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.442143 3954 consensus_queue.cc:260] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.443835 3954 ts_tablet_manager.cc:1428] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250622 13:52:56.443869 3955 ts_tablet_manager.cc:1428] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250622 13:52:56.444172 3954 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.444169 3955 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.446112 3955 tablet_bootstrap.cc:654] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.446112 3954 tablet_bootstrap.cc:654] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.448519 3953 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.448679 3953 ts_tablet_manager.cc:1397] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250622 13:52:56.449541 3953 raft_consensus.cc:357] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.449766 3953 raft_consensus.cc:383] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.449855 3953 raft_consensus.cc:738] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.450270 3953 consensus_queue.cc:260] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.450840 3953 ts_tablet_manager.cc:1428] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:56.451606 3953 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.452581 3954 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.452792 3954 ts_tablet_manager.cc:1397] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20250622 13:52:56.453502 3953 tablet_bootstrap.cc:654] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.453857 3954 raft_consensus.cc:357] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.454139 3954 raft_consensus.cc:383] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.454243 3954 raft_consensus.cc:738] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.454559 3954 consensus_queue.cc:260] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.456087 3954 ts_tablet_manager.cc:1428] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250622 13:52:56.456480 3954 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.456883 3953 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.457048 3953 ts_tablet_manager.cc:1397] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250622 13:52:56.457854 3953 raft_consensus.cc:357] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.458055 3953 raft_consensus.cc:383] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.458160 3953 raft_consensus.cc:738] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.458408 3953 consensus_queue.cc:260] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.458820 3954 tablet_bootstrap.cc:654] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.458928 3953 ts_tablet_manager.cc:1428] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:56.460048 3953 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.462007 3953 tablet_bootstrap.cc:654] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.461982 3955 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.462224 3955 ts_tablet_manager.cc:1397] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.012s
I20250622 13:52:56.462255 3954 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.462397 3954 ts_tablet_manager.cc:1397] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250622 13:52:56.463112 3955 raft_consensus.cc:357] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.463327 3955 raft_consensus.cc:383] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.463454 3955 raft_consensus.cc:738] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.463681 3955 consensus_queue.cc:260] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.464102 3953 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.464208 3955 ts_tablet_manager.cc:1428] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:52:56.464250 3953 ts_tablet_manager.cc:1397] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250622 13:52:56.464493 3955 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.464792 3954 raft_consensus.cc:357] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.464964 3954 raft_consensus.cc:383] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.465052 3954 raft_consensus.cc:738] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.465013 3953 raft_consensus.cc:357] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.465206 3953 raft_consensus.cc:383] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.465297 3953 raft_consensus.cc:738] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.465370 3954 consensus_queue.cc:260] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.465679 3953 consensus_queue.cc:260] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.466274 3953 ts_tablet_manager.cc:1428] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:56.466307 3955 tablet_bootstrap.cc:654] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.466507 3954 ts_tablet_manager.cc:1428] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250622 13:52:56.466627 3953 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.466812 3954 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.468806 3954 tablet_bootstrap.cc:654] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.469244 3953 tablet_bootstrap.cc:654] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.474632 3954 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.474869 3954 ts_tablet_manager.cc:1397] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250622 13:52:56.475965 3954 raft_consensus.cc:357] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.476271 3954 raft_consensus.cc:383] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.476382 3954 raft_consensus.cc:738] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.476776 3954 consensus_queue.cc:260] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.477439 3953 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.477604 3953 ts_tablet_manager.cc:1397] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20250622 13:52:56.477746 3954 ts_tablet_manager.cc:1428] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250622 13:52:56.478122 3954 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.478435 3953 raft_consensus.cc:357] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.478631 3953 raft_consensus.cc:383] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.478725 3953 raft_consensus.cc:738] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.479003 3953 consensus_queue.cc:260] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.479677 3953 ts_tablet_manager.cc:1428] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:56.480011 3953 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.480651 3954 tablet_bootstrap.cc:654] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.481900 3953 tablet_bootstrap.cc:654] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.483831 3959 raft_consensus.cc:491] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.483992 3959 raft_consensus.cc:513] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.484688 3961 raft_consensus.cc:491] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.485308 3955 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.485595 3955 ts_tablet_manager.cc:1397] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.011s
I20250622 13:52:56.486023 3959 leader_election.cc:290] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.486660 3955 raft_consensus.cc:357] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.486903 3955 raft_consensus.cc:383] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.486999 3955 raft_consensus.cc:738] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.487303 3955 consensus_queue.cc:260] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.484988 3961 raft_consensus.cc:513] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.497517 3961 leader_election.cc:290] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.505093 3955 ts_tablet_manager.cc:1428] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:52:56.509130 3954 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.509593 3954 ts_tablet_manager.cc:1397] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.012s
I20250622 13:52:56.510690 3954 raft_consensus.cc:357] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.511011 3954 raft_consensus.cc:383] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.509374 3955 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.512377 3953 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.512564 3953 ts_tablet_manager.cc:1397] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.011s
I20250622 13:52:56.513406 3955 tablet_bootstrap.cc:654] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.511715 3954 raft_consensus.cc:738] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.513847 3953 raft_consensus.cc:357] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.514436 3953 raft_consensus.cc:383] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.514518 3954 consensus_queue.cc:260] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.515270 3953 raft_consensus.cc:738] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.516140 3954 ts_tablet_manager.cc:1428] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250622 13:52:56.517329 3953 consensus_queue.cc:260] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.518183 3954 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.522758 3953 ts_tablet_manager.cc:1428] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20250622 13:52:56.523367 3961 raft_consensus.cc:491] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.523561 3961 raft_consensus.cc:513] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.525144 3953 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.525542 3961 leader_election.cc:290] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.535651 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
W20250622 13:52:56.536574 3800 leader_election.cc:343] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.537317 3953 tablet_bootstrap.cc:654] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.537881 3955 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.538116 3955 ts_tablet_manager.cc:1397] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.006s
I20250622 13:52:56.538549 3834 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
W20250622 13:52:56.539186 3724 leader_election.cc:343] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.539158 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
W20250622 13:52:56.539628 3724 leader_election.cc:343] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.539947 3955 raft_consensus.cc:357] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.540102 3955 raft_consensus.cc:383] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.540195 3955 raft_consensus.cc:738] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.540444 3955 consensus_queue.cc:260] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.540884 3955 ts_tablet_manager.cc:1428] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250622 13:52:56.541276 3955 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.543254 3955 tablet_bootstrap.cc:654] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.544061 3954 tablet_bootstrap.cc:654] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.547670 3953 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.547837 3953 ts_tablet_manager.cc:1397] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.001s
I20250622 13:52:56.548606 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.548679 3953 raft_consensus.cc:357] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.548868 3953 raft_consensus.cc:383] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.548729 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.548949 3953 raft_consensus.cc:738] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.549204 3953 consensus_queue.cc:260] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:52:56.549535 3724 leader_election.cc:343] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.549700 3724 leader_election.cc:304] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13e1e991fdc47899aac5b7c33fbbbd9; no voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:52:56.550266 3961 raft_consensus.cc:2747] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250622 13:52:56.552024 3955 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.552088 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.552203 3955 ts_tablet_manager.cc:1397] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
W20250622 13:52:56.552337 3724 leader_election.cc:343] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.549849 3953 ts_tablet_manager.cc:1428] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20250622 13:52:56.552935 3799 leader_election.cc:343] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.553074 3799 leader_election.cc:304] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:52:56.553336 3953 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.553511 3959 raft_consensus.cc:2747] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250622 13:52:56.553637 3955 raft_consensus.cc:357] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.553833 3955 raft_consensus.cc:383] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.553920 3955 raft_consensus.cc:738] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.552942 3724 leader_election.cc:304] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d13e1e991fdc47899aac5b7c33fbbbd9; no voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:52:56.554201 3955 consensus_queue.cc:260] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.554411 3961 raft_consensus.cc:2747] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250622 13:52:56.555958 3953 tablet_bootstrap.cc:654] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.557452 3955 ts_tablet_manager.cc:1428] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250622 13:52:56.558840 3954 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.559033 3954 ts_tablet_manager.cc:1397] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.015s
I20250622 13:52:56.559342 3955 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.560114 3954 raft_consensus.cc:357] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.560392 3954 raft_consensus.cc:383] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.560498 3954 raft_consensus.cc:738] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.560951 3955 tablet_bootstrap.cc:654] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.560832 3954 consensus_queue.cc:260] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.562494 3954 ts_tablet_manager.cc:1428] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250622 13:52:56.562876 3954 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.563796 3955 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.564303 3953 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.564482 3953 ts_tablet_manager.cc:1397] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250622 13:52:56.565331 3955 ts_tablet_manager.cc:1397] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250622 13:52:56.565284 3953 raft_consensus.cc:357] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.565541 3953 raft_consensus.cc:383] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.565637 3953 raft_consensus.cc:738] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.565732 3954 tablet_bootstrap.cc:654] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.565883 3953 consensus_queue.cc:260] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.566583 3955 raft_consensus.cc:357] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.566772 3955 raft_consensus.cc:383] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.566861 3955 raft_consensus.cc:738] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.567025 3953 ts_tablet_manager.cc:1428] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:56.567114 3955 consensus_queue.cc:260] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.567341 3953 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.569742 3953 tablet_bootstrap.cc:654] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.570885 3955 ts_tablet_manager.cc:1428] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250622 13:52:56.571230 3955 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.572314 3953 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.572541 3953 ts_tablet_manager.cc:1397] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250622 13:52:56.573650 3953 raft_consensus.cc:357] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.573805 3955 tablet_bootstrap.cc:654] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.573894 3953 raft_consensus.cc:383] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.573988 3953 raft_consensus.cc:738] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.574240 3953 consensus_queue.cc:260] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.574443 3954 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.574584 3954 ts_tablet_manager.cc:1397] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20250622 13:52:56.574909 3953 ts_tablet_manager.cc:1428] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:56.575174 3953 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.575335 3954 raft_consensus.cc:357] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.575511 3954 raft_consensus.cc:383] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.575594 3954 raft_consensus.cc:738] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.575845 3954 consensus_queue.cc:260] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.576819 3961 raft_consensus.cc:491] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.576982 3961 raft_consensus.cc:513] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.577450 3954 ts_tablet_manager.cc:1428] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250622 13:52:56.577642 3961 leader_election.cc:290] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.577728 3954 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.578248 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.579010 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.579205 3907 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 0.
I20250622 13:52:56.579252 3953 tablet_bootstrap.cc:654] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.579651 3724 leader_election.cc:304] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
W20250622 13:52:56.580078 3724 leader_election.cc:343] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.580461 3961 raft_consensus.cc:2802] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.580564 3961 raft_consensus.cc:491] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.580684 3961 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.580775 3954 tablet_bootstrap.cc:654] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.585099 3961 raft_consensus.cc:513] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.586457 3961 leader_election.cc:290] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.587110 3954 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.587121 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:56.587304 3954 ts_tablet_manager.cc:1397] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20250622 13:52:56.587311 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:56.587633 3907 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.588073 3953 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.588230 3953 ts_tablet_manager.cc:1397] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20250622 13:52:56.588227 3954 raft_consensus.cc:357] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.588430 3954 raft_consensus.cc:383] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.588536 3954 raft_consensus.cc:738] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.588775 3954 consensus_queue.cc:260] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.589555 3954 ts_tablet_manager.cc:1428] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:56.589682 3953 raft_consensus.cc:357] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:52:56.589776 3724 leader_election.cc:343] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.589843 3954 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.589890 3953 raft_consensus.cc:383] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.589965 3953 raft_consensus.cc:738] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.590178 3953 consensus_queue.cc:260] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.591342 3955 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.591475 3955 ts_tablet_manager.cc:1397] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.004s
I20250622 13:52:56.591769 3954 tablet_bootstrap.cc:654] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.592258 3955 raft_consensus.cc:357] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.592514 3955 raft_consensus.cc:383] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.592609 3955 raft_consensus.cc:738] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.592885 3955 consensus_queue.cc:260] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.593387 3955 ts_tablet_manager.cc:1428] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:52:56.593714 3955 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.593888 3954 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.594026 3954 ts_tablet_manager.cc:1397] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:52:56.594823 3954 raft_consensus.cc:357] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.595016 3954 raft_consensus.cc:383] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.595106 3954 raft_consensus.cc:738] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.595101 3907 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 1.
I20250622 13:52:56.595455 3954 consensus_queue.cc:260] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.595611 3724 leader_election.cc:304] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters: 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:52:56.595760 3953 ts_tablet_manager.cc:1428] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250622 13:52:56.596050 3953 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.596102 3961 raft_consensus.cc:2802] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.595633 3955 tablet_bootstrap.cc:654] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.599045 3953 tablet_bootstrap.cc:654] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.600744 3959 raft_consensus.cc:491] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.600942 3959 raft_consensus.cc:513] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.601806 3959 leader_election.cc:290] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.596056 3954 ts_tablet_manager.cc:1428] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:56.602394 3954 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.602597 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.602856 3759 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 0.
I20250622 13:52:56.603443 3800 leader_election.cc:304] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.604538 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.604884 3954 tablet_bootstrap.cc:654] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
W20250622 13:52:56.605286 3799 leader_election.cc:343] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.608376 3959 raft_consensus.cc:2802] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.608528 3959 raft_consensus.cc:491] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.608644 3959 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.612228 3954 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.612449 3954 ts_tablet_manager.cc:1397] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250622 13:52:56.615391 3961 raft_consensus.cc:695] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:52:56.615809 3961 consensus_queue.cc:237] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.617333 3954 raft_consensus.cc:357] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.617660 3954 raft_consensus.cc:383] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.617770 3954 raft_consensus.cc:738] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.618295 3959 raft_consensus.cc:513] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.618505 3954 consensus_queue.cc:260] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.619789 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.620050 3759 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.623939 3953 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.624070 3954 ts_tablet_manager.cc:1428] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250622 13:52:56.624104 3953 ts_tablet_manager.cc:1397] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.019s
I20250622 13:52:56.624284 3955 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.624414 3954 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Bootstrap starting.
I20250622 13:52:56.624441 3955 ts_tablet_manager.cc:1397] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.023s
I20250622 13:52:56.624828 3953 raft_consensus.cc:357] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.625006 3953 raft_consensus.cc:383] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.625090 3953 raft_consensus.cc:738] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.625262 3955 raft_consensus.cc:357] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.625404 3953 consensus_queue.cc:260] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.625515 3955 raft_consensus.cc:383] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.625685 3955 raft_consensus.cc:738] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.625922 3955 consensus_queue.cc:260] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.626020 3953 ts_tablet_manager.cc:1428] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:56.626353 3954 tablet_bootstrap.cc:654] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.626387 3953 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: Bootstrap starting.
I20250622 13:52:56.626453 3955 ts_tablet_manager.cc:1428] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:52:56.626752 3955 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.628688 3955 tablet_bootstrap.cc:654] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.628748 3953 tablet_bootstrap.cc:654] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.633952 3759 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 1.
I20250622 13:52:56.634899 3954 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: No bootstrap required, opened a new log
I20250622 13:52:56.635015 3800 leader_election.cc:304] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.635133 3954 ts_tablet_manager.cc:1397] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20250622 13:52:56.633105 3656 catalog_manager.cc:5582] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:56.635398 3996 raft_consensus.cc:2802] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.636721 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
W20250622 13:52:56.637516 3799 leader_election.cc:343] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): Illegal state: must be running to vote when last-logged opid is not known
I20250622 13:52:56.637811 3959 leader_election.cc:290] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.639755 3955 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.639915 3955 ts_tablet_manager.cc:1397] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20250622 13:52:56.640640 3955 raft_consensus.cc:357] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.640825 3955 raft_consensus.cc:383] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.643566 3954 raft_consensus.cc:357] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.643815 3954 raft_consensus.cc:383] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.643891 3954 raft_consensus.cc:738] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.644202 3954 consensus_queue.cc:260] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.644820 3954 ts_tablet_manager.cc:1428] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250622 13:52:56.646889 3955 raft_consensus.cc:738] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.647146 3955 consensus_queue.cc:260] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.647651 3955 ts_tablet_manager.cc:1428] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.008s user 0.008s sys 0.001s
I20250622 13:52:56.647902 3955 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.649327 3955 tablet_bootstrap.cc:654] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.649952 3953 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: No bootstrap required, opened a new log
I20250622 13:52:56.650118 3953 ts_tablet_manager.cc:1397] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.012s
I20250622 13:52:56.650467 3996 raft_consensus.cc:695] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:52:56.650719 3996 consensus_queue.cc:237] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.650873 3953 raft_consensus.cc:357] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.651027 3953 raft_consensus.cc:383] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.651083 3953 raft_consensus.cc:738] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.651316 3953 consensus_queue.cc:260] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.652735 3953 ts_tablet_manager.cc:1428] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:52:56.657037 3955 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.657182 3955 ts_tablet_manager.cc:1397] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.008s
I20250622 13:52:56.657824 3955 raft_consensus.cc:357] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.657956 3955 raft_consensus.cc:383] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.658075 3955 raft_consensus.cc:738] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.658285 3955 consensus_queue.cc:260] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.658717 3955 ts_tablet_manager.cc:1428] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250622 13:52:56.659055 3955 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.659685 3657 catalog_manager.cc:5582] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:56.660297 3955 tablet_bootstrap.cc:654] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.661762 3955 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.661890 3955 ts_tablet_manager.cc:1397] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250622 13:52:56.662518 3955 raft_consensus.cc:357] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.662680 3955 raft_consensus.cc:383] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.662731 3955 raft_consensus.cc:738] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.662959 3955 consensus_queue.cc:260] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.663491 3955 ts_tablet_manager.cc:1428] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:52:56.663720 3955 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: Bootstrap starting.
I20250622 13:52:56.665081 3955 tablet_bootstrap.cc:654] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: Neither blocks nor log segments found. Creating new log.
I20250622 13:52:56.666728 3955 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: No bootstrap required, opened a new log
I20250622 13:52:56.666826 3955 ts_tablet_manager.cc:1397] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:52:56.667277 3955 raft_consensus.cc:357] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.667407 3955 raft_consensus.cc:383] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:52:56.667452 3955 raft_consensus.cc:738] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Initialized, Role: FOLLOWER
I20250622 13:52:56.667640 3955 consensus_queue.cc:260] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.668004 3955 ts_tablet_manager.cc:1428] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:52:56.670071 3962 raft_consensus.cc:491] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.670215 3962 raft_consensus.cc:513] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.670928 3962 leader_election.cc:290] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.673835 3996 raft_consensus.cc:491] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.674011 3996 raft_consensus.cc:513] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.674618 3996 leader_election.cc:290] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649), d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517)
I20250622 13:52:56.675009 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8fa3d766949b99d0538f8b76498d5" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.675190 3907 raft_consensus.cc:2466] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 0.
I20250622 13:52:56.675503 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8fa3d766949b99d0538f8b76498d5" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.675652 3759 raft_consensus.cc:2466] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 0.
I20250622 13:52:56.676090 3800 leader_election.cc:304] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.676467 3996 raft_consensus.cc:2802] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.676535 3996 raft_consensus.cc:491] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.676601 3996 raft_consensus.cc:3058] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.676940 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.677063 3833 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.677402 3873 leader_election.cc:304] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:52:56.678792 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.678947 3759 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.679209 3996 raft_consensus.cc:513] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.679657 3996 leader_election.cc:290] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Requested vote from peers 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649), d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517)
I20250622 13:52:56.679832 3962 raft_consensus.cc:2802] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.679891 3962 raft_consensus.cc:491] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.679962 3962 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.680006 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8fa3d766949b99d0538f8b76498d5" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:56.680166 3907 raft_consensus.cc:3058] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.682636 3907 raft_consensus.cc:2466] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 1.
I20250622 13:52:56.682719 3962 raft_consensus.cc:513] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.682963 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8fa3d766949b99d0538f8b76498d5" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.683029 3799 leader_election.cc:304] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:52:56.683095 3759 raft_consensus.cc:3058] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.683329 3996 raft_consensus.cc:2802] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.683633 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.683706 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:56.683759 3758 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.683779 3996 raft_consensus.cc:695] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:52:56.683835 3833 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.684041 3996 consensus_queue.cc:237] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } }
I20250622 13:52:56.685318 3962 leader_election.cc:290] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.686550 3759 raft_consensus.cc:2466] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 1.
I20250622 13:52:56.686777 3758 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.687254 3874 leader_election.cc:304] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.687582 3833 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.687897 3962 raft_consensus.cc:2802] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.688331 3962 raft_consensus.cc:695] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:52:56.688647 3962 consensus_queue.cc:237] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.691638 3657 catalog_manager.cc:5582] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250622 13:52:56.691659 3656 catalog_manager.cc:5582] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:56.695847 3962 raft_consensus.cc:491] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.695987 3962 raft_consensus.cc:513] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.696530 3962 leader_election.cc:290] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.696924 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.696954 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.697077 3759 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.697090 3833 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.697451 3873 leader_election.cc:304] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:52:56.697674 3962 raft_consensus.cc:2802] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.697746 3962 raft_consensus.cc:491] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.697793 3962 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.699551 3962 raft_consensus.cc:513] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.700057 3962 leader_election.cc:290] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.700438 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.700455 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:56.700572 3759 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.700598 3833 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.702625 3759 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.702627 3833 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.703015 3874 leader_election.cc:304] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.703256 3962 raft_consensus.cc:2802] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.703382 3962 raft_consensus.cc:695] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:52:56.703616 3962 consensus_queue.cc:237] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.705907 3656 catalog_manager.cc:5582] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250622 13:52:56.709945 3962 raft_consensus.cc:491] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.710063 3962 raft_consensus.cc:513] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.710542 3962 leader_election.cc:290] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.710950 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.710960 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.711124 3759 raft_consensus.cc:2391] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3377fd0584c546c1a938e3a1204b2be3 in current term 1: Already voted for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in this term.
I20250622 13:52:56.711499 3874 leader_election.cc:304] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3; no voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:52:56.711724 3962 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.713781 3962 raft_consensus.cc:2747] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250622 13:52:56.714569 3996 raft_consensus.cc:491] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.714624 3962 raft_consensus.cc:491] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.714706 3962 raft_consensus.cc:513] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.714707 3996 raft_consensus.cc:513] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.715442 3996 leader_election.cc:290] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.715442 3962 leader_election.cc:290] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.715838 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.715837 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.715910 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.716040 3758 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 0.
I20250622 13:52:56.716040 3759 raft_consensus.cc:2466] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.716114 3833 raft_consensus.cc:2466] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.715838 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.716385 3800 leader_election.cc:304] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.716467 3907 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 0.
I20250622 13:52:56.716506 3874 leader_election.cc:304] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.716625 3962 raft_consensus.cc:491] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.716694 4008 raft_consensus.cc:2802] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.716780 4008 raft_consensus.cc:491] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.716867 4008 raft_consensus.cc:3058] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.716893 3996 raft_consensus.cc:2802] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.716959 3996 raft_consensus.cc:491] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.717034 3996 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.716734 3962 raft_consensus.cc:513] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.718201 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "677dddfcdf864e7f9b997d196b58bc78" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.718369 3758 raft_consensus.cc:2466] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.718686 3874 leader_election.cc:304] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.719029 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "677dddfcdf864e7f9b997d196b58bc78" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.719182 3833 raft_consensus.cc:2466] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.719583 4000 raft_consensus.cc:491] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.719676 3996 raft_consensus.cc:513] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.719691 4000 raft_consensus.cc:513] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.719980 4008 raft_consensus.cc:513] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.720247 4000 leader_election.cc:290] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.720494 3996 leader_election.cc:290] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.720558 4008 leader_election.cc:290] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.720701 4008 raft_consensus.cc:2802] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.720799 4008 raft_consensus.cc:491] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.720717 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.720876 4008 raft_consensus.cc:3058] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.720872 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.721004 3758 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.720789 3961 raft_consensus.cc:491] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.721078 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:56.721189 3833 raft_consensus.cc:3058] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.721156 3961 raft_consensus.cc:513] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.721668 3961 leader_election.cc:290] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.722000 3759 raft_consensus.cc:2466] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 0.
I20250622 13:52:56.722178 3834 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.722313 3834 raft_consensus.cc:2466] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 0.
I20250622 13:52:56.722656 3724 leader_election.cc:304] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.722738 3800 leader_election.cc:304] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.722896 3961 raft_consensus.cc:2802] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.722978 3961 raft_consensus.cc:491] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.723037 3961 raft_consensus.cc:3058] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.723305 3996 raft_consensus.cc:2802] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.723416 3996 raft_consensus.cc:491] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.723515 3996 raft_consensus.cc:3058] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.723626 3758 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 1.
I20250622 13:52:56.723905 3833 raft_consensus.cc:2466] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.724009 3800 leader_election.cc:304] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.724223 3873 leader_election.cc:304] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:52:56.724258 4000 raft_consensus.cc:2802] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.724413 4000 raft_consensus.cc:695] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:52:56.724720 3962 leader_election.cc:290] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.724658 4000 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.724858 3962 raft_consensus.cc:2802] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.724957 3962 raft_consensus.cc:695] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:52:56.725198 3962 consensus_queue.cc:237] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.725608 3961 raft_consensus.cc:513] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.726644 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.726837 3907 raft_consensus.cc:2466] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 0.
I20250622 13:52:56.727283 3996 raft_consensus.cc:513] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.727963 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:56.728138 3907 raft_consensus.cc:3058] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.728431 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.728765 3758 raft_consensus.cc:2391] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in current term 1: Already voted for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in this term.
I20250622 13:52:56.729547 3996 leader_election.cc:290] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.731693 3907 raft_consensus.cc:2466] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 1.
I20250622 13:52:56.732115 3961 leader_election.cc:290] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.732149 3724 leader_election.cc:304] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.732518 3961 raft_consensus.cc:2802] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.732717 3961 raft_consensus.cc:695] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:52:56.733031 3961 consensus_queue.cc:237] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.734884 4008 raft_consensus.cc:513] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.735284 3656 catalog_manager.cc:5582] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250622 13:52:56.732475 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:56.735962 3833 raft_consensus.cc:2391] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d13e1e991fdc47899aac5b7c33fbbbd9 in current term 1: Already voted for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in this term.
I20250622 13:52:56.736088 3834 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "677dddfcdf864e7f9b997d196b58bc78" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:56.736299 3834 raft_consensus.cc:3058] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.737002 4008 leader_election.cc:290] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.738606 3961 raft_consensus.cc:491] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.738795 3961 raft_consensus.cc:513] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.739430 3961 leader_election.cc:290] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.739835 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.740007 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.740233 3907 raft_consensus.cc:2466] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 0.
I20250622 13:52:56.740028 3834 raft_consensus.cc:2466] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.740181 3657 catalog_manager.cc:5582] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:56.740031 3833 raft_consensus.cc:2466] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 0.
I20250622 13:52:56.740715 3724 leader_election.cc:304] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.740841 3873 leader_election.cc:304] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:52:56.741150 3962 raft_consensus.cc:2802] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.741150 3961 raft_consensus.cc:2802] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.741329 3961 raft_consensus.cc:491] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.741380 3962 raft_consensus.cc:695] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:52:56.741719 3962 consensus_queue.cc:237] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.743335 3962 raft_consensus.cc:491] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.743551 3962 raft_consensus.cc:513] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.744982 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91736fef6df543c4953497186dfe1c5e" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" is_pre_election: true
I20250622 13:52:56.744944 3656 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:56.745260 3833 raft_consensus.cc:2466] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.745406 3654 catalog_manager.cc:5582] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250622 13:52:56.745839 3873 leader_election.cc:304] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:52:56.746169 3962 leader_election.cc:290] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.746405 3962 raft_consensus.cc:2802] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.746529 3962 raft_consensus.cc:491] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.746690 3962 raft_consensus.cc:3058] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.749116 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.749327 3758 raft_consensus.cc:3058] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.749478 3961 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.750092 3962 raft_consensus.cc:513] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.750924 3962 leader_election.cc:290] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:56.751427 3833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91736fef6df543c4953497186dfe1c5e" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:56.751652 3833 raft_consensus.cc:3058] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.752135 3961 raft_consensus.cc:513] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.752648 3961 leader_election.cc:290] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.753123 3834 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:56.753296 3834 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.753825 3758 raft_consensus.cc:2466] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.755003 3833 raft_consensus.cc:2466] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.755125 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "677dddfcdf864e7f9b997d196b58bc78" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.755327 3758 raft_consensus.cc:3058] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.755379 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:56.755506 3873 leader_election.cc:304] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:52:56.755565 3907 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.756505 3759 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91736fef6df543c4953497186dfe1c5e" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.756594 3834 raft_consensus.cc:2466] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 1.
I20250622 13:52:56.756716 3759 raft_consensus.cc:2466] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 0.
I20250622 13:52:56.757007 3724 leader_election.cc:304] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.757306 3961 raft_consensus.cc:2802] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.757300 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.757632 3961 raft_consensus.cc:695] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:52:56.757719 3908 raft_consensus.cc:2391] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in current term 1: Already voted for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in this term.
I20250622 13:52:56.758013 3961 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.758235 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:56.758445 3908 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.758930 3758 raft_consensus.cc:2466] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.758977 3907 raft_consensus.cc:2466] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 1.
I20250622 13:52:56.759416 3962 raft_consensus.cc:2802] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.760857 3962 raft_consensus.cc:695] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:52:56.761092 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91736fef6df543c4953497186dfe1c5e" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.761281 3758 raft_consensus.cc:3058] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.761605 3907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:56.761828 3907 raft_consensus.cc:2391] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in current term 1: Already voted for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in this term.
I20250622 13:52:56.761895 3656 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:56.762360 3908 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 1.
I20250622 13:52:56.763228 3799 leader_election.cc:304] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:52:56.764298 3758 raft_consensus.cc:2466] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 1.
I20250622 13:52:56.764609 4000 raft_consensus.cc:2747] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250622 13:52:56.761188 3962 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.767261 3656 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250622 13:52:56.854321 4000 raft_consensus.cc:491] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.854586 4000 raft_consensus.cc:513] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.855268 4000 leader_election.cc:290] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.855782 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.855822 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.856004 3908 raft_consensus.cc:2466] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 0.
I20250622 13:52:56.856002 3758 raft_consensus.cc:2466] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 0.
I20250622 13:52:56.856387 3799 leader_election.cc:304] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:52:56.856642 4000 raft_consensus.cc:2802] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:52:56.856720 4000 raft_consensus.cc:491] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:52:56.856791 4000 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.858682 4000 raft_consensus.cc:513] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.859221 4000 leader_election.cc:290] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.859652 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:52:56.859716 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:56.859858 3758 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.859866 3908 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.862370 3908 raft_consensus.cc:2466] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 1.
I20250622 13:52:56.862365 3758 raft_consensus.cc:2466] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 1.
I20250622 13:52:56.862758 3800 leader_election.cc:304] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:56.863018 4000 raft_consensus.cc:2802] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:52:56.863181 4000 raft_consensus.cc:695] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:52:56.863466 4000 consensus_queue.cc:237] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.865871 3656 catalog_manager.cc:5582] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:56.960129 3908 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:56.960326 3834 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:52:56.961185 3991 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:56.963565 3833 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:56.964496 3834 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:56.966110 3908 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:56.966883 3991 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:56.967585 3991 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:56.968320 3961 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:56.969419 4000 raft_consensus.cc:491] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:52:56.969578 4000 raft_consensus.cc:513] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:56.970293 4000 leader_election.cc:290] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:56.970880 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3377fd0584c546c1a938e3a1204b2be3" is_pre_election: true
I20250622 13:52:56.970906 3758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" is_pre_election: true
I20250622 13:52:56.973726 3832 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:56.974123 3799 leader_election.cc:304] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:52:56.974249 3961 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:56.974462 3908 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:56.975844 3991 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:56.987495 3831 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:56.989698 3962 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.016311 3830 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.017215 3758 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.017874 4008 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.018378 3758 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.018440 3830 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.018904 4008 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.018945 4012 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.020031 4012 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.027884 3757 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.028698 4012 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.064472 3756 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.065398 4008 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.085230 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:57.090876 3831 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.108173 3832 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.111219 3755 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.116981 4012 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20250622 13:52:57.145102 4008 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20250622 13:52:57.172334 4066 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20250622 13:52:57.185226 3754 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.190421 3996 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250622 13:52:57.200611 3996 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
W20250622 13:52:57.201326 4018 log.cc:927] Time spent T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250622 13:52:57.209678 3996 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.222152 3996 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250622 13:52:57.222772 3905 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.265820 3753 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:52:57.266705 3959 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:52:57.275547 3758 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:52:57.283236 3996 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20250622 13:52:57.292104 4000 raft_consensus.cc:2747] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250622 13:52:57.294999 3959 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
W20250622 13:52:57.321391 3724 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.67:33649, user_credentials={real_user=slave}} blocked reactor thread for 213155us
I20250622 13:52:57.349521 3903 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250622 13:52:57.364089 4062 log.cc:927] Time spent T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20250622 13:52:57.391700 4071 mvcc.cc:204] Tried to move back new op lower bound from 7170459144260706304 to 7170459143230611456. Current Snapshot: MvccSnapshot[applied={T|T < 7170459144260706304}]
W20250622 13:52:57.426481 4051 log.cc:927] Time spent T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20250622 13:52:57.427282 4073 log.cc:927] Time spent T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20250622 13:52:57.427321 4081 log.cc:927] Time spent T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250622 13:52:57.442356 4079 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
W20250622 13:52:57.569406 4072 log.cc:927] Time spent T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20250622 13:52:57.586040 3874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.65:40517, user_credentials={real_user=slave}} blocked reactor thread for 117724us
I20250622 13:52:57.696133 3904 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:52:57.696398 3904 raft_consensus.cc:604] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:52:57.702759 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:52:57.704879 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250622 13:52:57.765342 3887 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.797250 3885 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.801445 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.818291 3888 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.831221 3888 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.838088 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.844226 3888 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.855834 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.864315 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.865150 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.881433 3888 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.882165 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.883299 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.891459 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.895404 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.896513 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.899166 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.900521 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.905578 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.906978 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.908264 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.910838 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.913975 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.918555 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.920056 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.921347 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.922689 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.929384 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.931910 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.933130 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.934361 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.943095 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.943207 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.944633 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.945791 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.956442 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.957686 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.960009 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.960144 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.969933 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.969998 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.973824 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.973887 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.983770 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.986361 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.986390 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:57.988790 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.000651 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.000653 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.001916 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.003053 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.016775 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.018096 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.018095 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.022604 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.035396 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.035395 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.036741 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.041112 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.051831 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.053226 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.053170 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.060616 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.071799 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.071799 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.074507 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.080021 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.092021 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.092021 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.093498 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.103250 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.114261 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.114261 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.115432 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.124122 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.137054 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.137076 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.138255 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.145846 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.158772 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.158805 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.162303 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.168921 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.184672 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.185847 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.188205 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.196834 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.208639 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.213479 3883 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.213584 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.223707 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.236521 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.237596 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.240878 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.249282 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.262858 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.263949 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.269296 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
W20250622 13:52:58.280973 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
I20250622 13:52:58.289794 4012 raft_consensus.cc:991] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:52:58.289943 4012 raft_consensus.cc:1079] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Signalling peer d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:52:58.290575 3758 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:53794
W20250622 13:52:58.290639 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
I20250622 13:52:58.290766 3758 raft_consensus.cc:491] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:52:58.290882 3758 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:52:58.292909 3758 raft_consensus.cc:513] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:52:58.293124 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Service unavailable: leader transfer in progress
I20250622 13:52:58.293540 3758 leader_election.cc:290] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 2 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:58.293864 3830 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:58.293952 3904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:58.294016 3830 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:52:58.294126 3904 raft_consensus.cc:3053] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:52:58.294190 3904 raft_consensus.cc:738] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:52:58.294401 3904 consensus_queue.cc:260] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:58.294664 3904 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:52:58.296299 3830 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 2.
I20250622 13:52:58.296691 3724 leader_election.cc:304] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:58.296923 4083 raft_consensus.cc:2802] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:52:58.297024 4083 raft_consensus.cc:695] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:52:58.297199 3904 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 2.
I20250622 13:52:58.297251 4083 consensus_queue.cc:237] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:52:58.298813 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Illegal state: replica 3377fd0584c546c1a938e3a1204b2be3 is not leader of this config: current role FOLLOWER
I20250622 13:52:58.299801 3656 catalog_manager.cc:5582] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 1 to 2, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
W20250622 13:52:58.310467 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Illegal state: replica 3377fd0584c546c1a938e3a1204b2be3 is not leader of this config: current role FOLLOWER
W20250622 13:52:58.318115 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Illegal state: replica 3377fd0584c546c1a938e3a1204b2be3 is not leader of this config: current role FOLLOWER
W20250622 13:52:58.320485 3884 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36358: Illegal state: replica 3377fd0584c546c1a938e3a1204b2be3 is not leader of this config: current role FOLLOWER
I20250622 13:52:58.330168 3904 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250622 13:52:58.330160 3830 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250622 13:52:58.330721 4078 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:52:58.330895 4083 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250622 13:52:58.497539 3724 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:34391, user_credentials={real_user=slave}} blocked reactor thread for 67008.1us
W20250622 13:52:58.912289 3874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.65:40517, user_credentials={real_user=slave}} blocked reactor thread for 52847.4us
I20250622 13:52:58.950469 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:58.956636 4138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:58.961827 4139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:58.962582 4141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:58.966470 413 server_base.cc:1048] running on GCE node
I20250622 13:52:58.966707 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:58.966770 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:58.966810 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600378966807 us; error 0 us; skew 500 ppm
I20250622 13:52:58.966948 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:58.976024 413 webserver.cc:469] Webserver started at http://127.0.103.68:43349/ using document root <none> and password file <none>
I20250622 13:52:58.976300 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:58.976366 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:58.976501 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:58.977180 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-3-root/instance:
uuid: "d23b47356efa46c792ab125b568ea95a"
format_stamp: "Formatted at 2025-06-22 13:52:58 on dist-test-slave-tmv0"
I20250622 13:52:58.979703 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250622 13:52:58.981284 4146 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:58.981669 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250622 13:52:58.981781 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-3-root
uuid: "d23b47356efa46c792ab125b568ea95a"
format_stamp: "Formatted at 2025-06-22 13:52:58 on dist-test-slave-tmv0"
I20250622 13:52:58.981899 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:59.008965 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:59.009565 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:59.010102 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:59.010962 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:59.011021 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:59.011098 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:59.011142 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:59.027344 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:33811
I20250622 13:52:59.027374 4208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:33811 every 8 connection(s)
I20250622 13:52:59.029829 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:59.033227 4213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:59.033203 4214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:59.037564 4209 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37277
I20250622 13:52:59.037765 4209 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:59.038349 4209 heartbeater.cc:507] Master 127.0.103.126:37277 requested a full tablet report, sending...
I20250622 13:52:59.039331 3656 ts_manager.cc:194] Registered new tserver with Master: d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
W20250622 13:52:59.040017 4217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:59.040055 413 server_base.cc:1048] running on GCE node
I20250622 13:52:59.040314 3656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58414
I20250622 13:52:59.040373 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:59.040418 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:59.040460 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600379040456 us; error 0 us; skew 500 ppm
I20250622 13:52:59.040596 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:59.042093 413 webserver.cc:469] Webserver started at http://127.0.103.69:41883/ using document root <none> and password file <none>
I20250622 13:52:59.042292 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:59.042346 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:59.042434 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:59.042953 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-4-root/instance:
uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
format_stamp: "Formatted at 2025-06-22 13:52:59 on dist-test-slave-tmv0"
I20250622 13:52:59.045255 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:52:59.046826 4221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:59.047163 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250622 13:52:59.047262 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-4-root
uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
format_stamp: "Formatted at 2025-06-22 13:52:59 on dist-test-slave-tmv0"
I20250622 13:52:59.047358 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:59.055274 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:59.055799 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:59.056277 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:59.057683 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:59.057744 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:59.057840 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:59.057884 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:59.076272 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:42925
I20250622 13:52:59.076301 4283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:42925 every 8 connection(s)
I20250622 13:52:59.078059 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:52:59.081339 4287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:52:59.082715 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:59.083613 413 server_base.cc:1048] running on GCE node
W20250622 13:52:59.084460 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:52:59.084857 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:52:59.084955 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:52:59.085021 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600379084999 us; error 0 us; skew 500 ppm
I20250622 13:52:59.085220 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:52:59.086126 4284 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37277
I20250622 13:52:59.086241 4284 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:59.086638 4284 heartbeater.cc:507] Master 127.0.103.126:37277 requested a full tablet report, sending...
I20250622 13:52:59.086916 413 webserver.cc:469] Webserver started at http://127.0.103.70:43913/ using document root <none> and password file <none>
I20250622 13:52:59.087157 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:52:59.087242 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:52:59.087394 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:52:59.087424 3656 ts_manager.cc:194] Registered new tserver with Master: feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925)
I20250622 13:52:59.088052 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-5-root/instance:
uuid: "52c21167280a4d669123bb1ed7675d79"
format_stamp: "Formatted at 2025-06-22 13:52:59 on dist-test-slave-tmv0"
I20250622 13:52:59.088255 3656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58424
I20250622 13:52:59.090651 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20250622 13:52:59.092257 4295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:59.092635 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:52:59.092763 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-5-root
uuid: "52c21167280a4d669123bb1ed7675d79"
format_stamp: "Formatted at 2025-06-22 13:52:59 on dist-test-slave-tmv0"
I20250622 13:52:59.092861 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:52:59.112424 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:52:59.112943 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:52:59.113544 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:52:59.114562 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:52:59.114648 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:59.114741 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:52:59.114794 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:52:59.132295 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.70:35833
I20250622 13:52:59.132368 4357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.70:35833 every 8 connection(s)
I20250622 13:52:59.140277 4358 heartbeater.cc:344] Connected to a master server at 127.0.103.126:37277
I20250622 13:52:59.140435 4358 heartbeater.cc:461] Registering TS with master...
I20250622 13:52:59.140934 4358 heartbeater.cc:507] Master 127.0.103.126:37277 requested a full tablet report, sending...
I20250622 13:52:59.141896 3656 ts_manager.cc:194] Registered new tserver with Master: 52c21167280a4d669123bb1ed7675d79 (127.0.103.70:35833)
I20250622 13:52:59.142663 3656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58426
I20250622 13:52:59.353000 3908 consensus_queue.cc:237] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:52:59.355032 3829 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.355712 4095 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.356508 3756 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.357132 4095 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.358131 4137 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:52:59.359055 3829 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:52:59.362962 3756 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
W20250622 13:52:59.365293 3874 consensus_peers.cc:487] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 -> Peer feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925): Couldn't send request to peer feaebd1ce3104a2fa3cc207fd3c23468. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d317ca8305314d45b47fa1014f3b5162. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:52:59.365960 3656 catalog_manager.cc:5582] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:52:59.376744 3829 consensus_queue.cc:237] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:52:59.378883 3908 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.379235 3756 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.379565 4105 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.379917 3996 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.382946 3959 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:52:59.383270 3908 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:52:59.383894 3756 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:52:59.389460 3652 catalog_manager.cc:5582] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index -1 to 28, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:59.390268 3829 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
W20250622 13:52:59.392832 3798 consensus_peers.cc:487] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 -> Peer 52c21167280a4d669123bb1ed7675d79 (127.0.103.70:35833): Couldn't send request to peer 52c21167280a4d669123bb1ed7675d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1aa8fa3d766949b99d0538f8b76498d5. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:52:59.393287 3908 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.393790 3756 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.393927 3959 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.394433 4000 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.397105 4000 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:52:59.401548 3753 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
W20250622 13:52:59.401954 3797 consensus_peers.cc:487] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 -> Peer d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811): Couldn't send request to peer d23b47356efa46c792ab125b568ea95a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9260a075f104769bc81ca7eb0c76cb2. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:52:59.402770 3908 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:52:59.405872 3829 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.406383 3903 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:52:59.407410 3651 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:52:59.406505 4137 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.410141 3753 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.412042 4137 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20250622 13:52:59.412962 3872 consensus_peers.cc:487] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 -> Peer 52c21167280a4d669123bb1ed7675d79 (127.0.103.70:35833): Couldn't send request to peer 52c21167280a4d669123bb1ed7675d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91736fef6df543c4953497186dfe1c5e. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:52:59.413369 4114 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:52:59.413898 3829 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:52:59.415475 3756 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:52:59.419389 3651 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index -1 to 28, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:59.419742 3903 consensus_queue.cc:237] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:52:59.422567 3829 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.422567 3756 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20250622 13:52:59.422597 3874 consensus_peers.cc:487] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 -> Peer feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925): Couldn't send request to peer feaebd1ce3104a2fa3cc207fd3c23468. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 677dddfcdf864e7f9b997d196b58bc78. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:52:59.423210 4137 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.423714 4137 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.426123 4137 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:52:59.426757 3756 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:52:59.427012 3829 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:52:59.431226 3652 catalog_manager.cc:5582] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:52:59.442836 3756 consensus_queue.cc:237] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:52:59.446147 3829 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.446367 3903 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:52:59.446920 4083 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.447672 4083 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:52:59.449642 4119 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:52:59.454316 3903 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:52:59.456596 3656 catalog_manager.cc:5582] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:59.454459 3829 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
W20250622 13:52:59.459100 3722 consensus_peers.cc:487] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 -> Peer d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811): Couldn't send request to peer d23b47356efa46c792ab125b568ea95a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f10daa2b314a4660bd36fa7902789a8e. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:52:59.705245 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:52:59.708215 3903 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:52:59.708434 3903 raft_consensus.cc:604] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:52:59.708917 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:52:59.709106 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:52:59.845340 4375 ts_tablet_manager.cc:927] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: Initiating tablet copy from peer 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:59.846024 4375 tablet_copy_client.cc:323] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:34391
I20250622 13:52:59.851722 3844 tablet_copy_service.cc:140] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Received BeginTabletCopySession request for tablet c9260a075f104769bc81ca7eb0c76cb2 from peer d23b47356efa46c792ab125b568ea95a ({username='slave'} at 127.0.0.1:45644)
I20250622 13:52:59.851917 3844 tablet_copy_service.cc:161] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Beginning new tablet copy session on tablet c9260a075f104769bc81ca7eb0c76cb2 from peer d23b47356efa46c792ab125b568ea95a at {username='slave'} at 127.0.0.1:45644: session id = d23b47356efa46c792ab125b568ea95a-c9260a075f104769bc81ca7eb0c76cb2
I20250622 13:52:59.854561 4379 ts_tablet_manager.cc:927] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: Initiating tablet copy from peer 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:52:59.854779 3844 tablet_copy_source_session.cc:215] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:52:59.855237 4379 tablet_copy_client.cc:323] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:34391
I20250622 13:52:59.856441 4375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9260a075f104769bc81ca7eb0c76cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:59.865190 4375 tablet_copy_client.cc:806] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 0 data blocks...
I20250622 13:52:59.865566 4375 tablet_copy_client.cc:670] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 1 WAL segments...
I20250622 13:52:59.866848 4378 ts_tablet_manager.cc:927] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: Initiating tablet copy from peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:59.867285 4376 tablet_copy_client.cc:1025] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: tablet copy: Injecting latency into file download: 2000
I20250622 13:52:59.867502 4378 tablet_copy_client.cc:323] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33649
I20250622 13:52:59.875339 3844 tablet_copy_service.cc:140] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Received BeginTabletCopySession request for tablet 1aa8fa3d766949b99d0538f8b76498d5 from peer 52c21167280a4d669123bb1ed7675d79 ({username='slave'} at 127.0.0.1:45650)
I20250622 13:52:59.875612 3844 tablet_copy_service.cc:161] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Beginning new tablet copy session on tablet 1aa8fa3d766949b99d0538f8b76498d5 from peer 52c21167280a4d669123bb1ed7675d79 at {username='slave'} at 127.0.0.1:45650: session id = 52c21167280a4d669123bb1ed7675d79-1aa8fa3d766949b99d0538f8b76498d5
I20250622 13:52:59.880308 3844 tablet_copy_source_session.cc:215] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:52:59.881793 4379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa8fa3d766949b99d0538f8b76498d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:59.887614 3918 tablet_copy_service.cc:140] P 3377fd0584c546c1a938e3a1204b2be3: Received BeginTabletCopySession request for tablet 91736fef6df543c4953497186dfe1c5e from peer 52c21167280a4d669123bb1ed7675d79 ({username='slave'} at 127.0.0.1:36376)
I20250622 13:52:59.887848 3918 tablet_copy_service.cc:161] P 3377fd0584c546c1a938e3a1204b2be3: Beginning new tablet copy session on tablet 91736fef6df543c4953497186dfe1c5e from peer 52c21167280a4d669123bb1ed7675d79 at {username='slave'} at 127.0.0.1:36376: session id = 52c21167280a4d669123bb1ed7675d79-91736fef6df543c4953497186dfe1c5e
I20250622 13:52:59.889230 4379 tablet_copy_client.cc:806] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 0 data blocks...
I20250622 13:52:59.889520 4379 tablet_copy_client.cc:670] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 1 WAL segments...
I20250622 13:52:59.890341 3918 tablet_copy_source_session.cc:215] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:52:59.891077 4380 tablet_copy_client.cc:1025] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Injecting latency into file download: 2000
I20250622 13:52:59.891813 4378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91736fef6df543c4953497186dfe1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:59.898041 4114 raft_consensus.cc:991] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:52:59.898286 4114 raft_consensus.cc:1079] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Signalling peer d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:52:59.900172 4378 tablet_copy_client.cc:806] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 0 data blocks...
I20250622 13:52:59.900506 3753 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:53794
I20250622 13:52:59.902287 4378 tablet_copy_client.cc:670] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 1 WAL segments...
I20250622 13:52:59.902658 3753 raft_consensus.cc:491] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:52:59.902859 3753 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:52:59.906706 3753 raft_consensus.cc:513] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:59.907411 3753 leader_election.cc:290] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 2 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:59.907951 3829 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:52:59.907999 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:52:59.908116 3829 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:52:59.908172 3908 raft_consensus.cc:3053] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:52:59.908272 3908 raft_consensus.cc:738] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:52:59.908519 3908 consensus_queue.cc:260] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:59.908963 3908 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:52:59.911343 3829 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 2.
I20250622 13:52:59.911716 4382 tablet_copy_client.cc:1025] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: tablet copy: Injecting latency into file download: 2000
I20250622 13:52:59.911752 3724 leader_election.cc:304] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:52:59.913223 3908 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 2.
I20250622 13:52:59.913594 4083 raft_consensus.cc:2802] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:52:59.913760 4083 raft_consensus.cc:695] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:52:59.914038 4083 consensus_queue.cc:237] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:52:59.916543 3656 catalog_manager.cc:5582] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 1 to 2, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:52:59.935714 4384 ts_tablet_manager.cc:927] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: Initiating tablet copy from peer d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517)
I20250622 13:52:59.936295 4384 tablet_copy_client.cc:323] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:40517
I20250622 13:52:59.941969 3769 tablet_copy_service.cc:140] P d13e1e991fdc47899aac5b7c33fbbbd9: Received BeginTabletCopySession request for tablet f10daa2b314a4660bd36fa7902789a8e from peer d23b47356efa46c792ab125b568ea95a ({username='slave'} at 127.0.0.1:53832)
I20250622 13:52:59.942147 3769 tablet_copy_service.cc:161] P d13e1e991fdc47899aac5b7c33fbbbd9: Beginning new tablet copy session on tablet f10daa2b314a4660bd36fa7902789a8e from peer d23b47356efa46c792ab125b568ea95a at {username='slave'} at 127.0.0.1:53832: session id = d23b47356efa46c792ab125b568ea95a-f10daa2b314a4660bd36fa7902789a8e
I20250622 13:52:59.944278 3769 tablet_copy_source_session.cc:215] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:52:59.945485 4384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10daa2b314a4660bd36fa7902789a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:59.950874 4384 tablet_copy_client.cc:806] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 0 data blocks...
I20250622 13:52:59.951125 4384 tablet_copy_client.cc:670] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 1 WAL segments...
I20250622 13:52:59.952358 4385 tablet_copy_client.cc:1025] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: tablet copy: Injecting latency into file download: 2000
I20250622 13:52:59.970539 4386 ts_tablet_manager.cc:927] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: Initiating tablet copy from peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:52:59.971110 4386 tablet_copy_client.cc:323] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33649
I20250622 13:52:59.976873 3918 tablet_copy_service.cc:140] P 3377fd0584c546c1a938e3a1204b2be3: Received BeginTabletCopySession request for tablet d317ca8305314d45b47fa1014f3b5162 from peer feaebd1ce3104a2fa3cc207fd3c23468 ({username='slave'} at 127.0.0.1:36388)
I20250622 13:52:59.977025 3918 tablet_copy_service.cc:161] P 3377fd0584c546c1a938e3a1204b2be3: Beginning new tablet copy session on tablet d317ca8305314d45b47fa1014f3b5162 from peer feaebd1ce3104a2fa3cc207fd3c23468 at {username='slave'} at 127.0.0.1:36388: session id = feaebd1ce3104a2fa3cc207fd3c23468-d317ca8305314d45b47fa1014f3b5162
I20250622 13:52:59.979712 3918 tablet_copy_source_session.cc:215] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:52:59.981033 4386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d317ca8305314d45b47fa1014f3b5162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:52:59.986673 4386 tablet_copy_client.cc:806] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 0 data blocks...
I20250622 13:52:59.986932 4386 tablet_copy_client.cc:670] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 1 WAL segments...
I20250622 13:52:59.988389 4387 tablet_copy_client.cc:1025] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:00.024924 4389 ts_tablet_manager.cc:927] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: Initiating tablet copy from peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:00.025628 4389 tablet_copy_client.cc:323] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33649
I20250622 13:53:00.026157 3918 tablet_copy_service.cc:140] P 3377fd0584c546c1a938e3a1204b2be3: Received BeginTabletCopySession request for tablet 677dddfcdf864e7f9b997d196b58bc78 from peer feaebd1ce3104a2fa3cc207fd3c23468 ({username='slave'} at 127.0.0.1:36388)
I20250622 13:53:00.026275 3918 tablet_copy_service.cc:161] P 3377fd0584c546c1a938e3a1204b2be3: Beginning new tablet copy session on tablet 677dddfcdf864e7f9b997d196b58bc78 from peer feaebd1ce3104a2fa3cc207fd3c23468 at {username='slave'} at 127.0.0.1:36388: session id = feaebd1ce3104a2fa3cc207fd3c23468-677dddfcdf864e7f9b997d196b58bc78
I20250622 13:53:00.028458 3918 tablet_copy_source_session.cc:215] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:00.029553 4389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 677dddfcdf864e7f9b997d196b58bc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:00.034050 4389 tablet_copy_client.cc:806] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:00.034253 4389 tablet_copy_client.cc:670] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:00.035568 4390 tablet_copy_client.cc:1025] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:00.041517 4209 heartbeater.cc:499] Master 127.0.103.126:37277 was elected leader, sending a full tablet report...
I20250622 13:53:00.089478 4284 heartbeater.cc:499] Master 127.0.103.126:37277 was elected leader, sending a full tablet report...
I20250622 13:53:00.143854 4358 heartbeater.cc:499] Master 127.0.103.126:37277 was elected leader, sending a full tablet report...
I20250622 13:53:00.366741 3829 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250622 13:53:00.367265 4119 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:00.382504 3908 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250622 13:53:00.383001 4397 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:01.709494 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:01.712082 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:01.712281 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:01.867693 4375 tablet_copy_client.cc:538] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:01.870803 4375 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: Bootstrap starting.
I20250622 13:53:01.891484 4379 tablet_copy_client.cc:538] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:01.894770 4379 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: Bootstrap starting.
I20250622 13:53:01.912124 4378 tablet_copy_client.cc:538] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:01.915550 4378 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: Bootstrap starting.
I20250622 13:53:01.918565 4375 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:01.919427 4375 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: Bootstrap complete.
I20250622 13:53:01.919780 4375 ts_tablet_manager.cc:1397] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: Time spent bootstrapping tablet: real 0.049s user 0.042s sys 0.001s
I20250622 13:53:01.921029 4375 raft_consensus.cc:357] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:01.921391 4375 raft_consensus.cc:738] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d23b47356efa46c792ab125b568ea95a, State: Initialized, Role: LEARNER
I20250622 13:53:01.921662 4375 consensus_queue.cc:260] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:01.933791 4375 ts_tablet_manager.cc:1428] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: Time spent starting tablet: real 0.014s user 0.007s sys 0.003s
I20250622 13:53:01.934482 3844 tablet_copy_service.cc:342] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Request end of tablet copy session d23b47356efa46c792ab125b568ea95a-c9260a075f104769bc81ca7eb0c76cb2 received from {username='slave'} at 127.0.0.1:45644
I20250622 13:53:01.934587 3844 tablet_copy_service.cc:434] P 946d416d6f4a4b0ea6a77b93cd6f58f6: ending tablet copy session d23b47356efa46c792ab125b568ea95a-c9260a075f104769bc81ca7eb0c76cb2 on tablet c9260a075f104769bc81ca7eb0c76cb2 with peer d23b47356efa46c792ab125b568ea95a
I20250622 13:53:01.944841 4379 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:01.945223 4379 tablet_bootstrap.cc:492] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: Bootstrap complete.
I20250622 13:53:01.945528 4379 ts_tablet_manager.cc:1397] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: Time spent bootstrapping tablet: real 0.051s user 0.047s sys 0.001s
I20250622 13:53:01.946143 4379 raft_consensus.cc:357] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:01.946298 4379 raft_consensus.cc:738] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 52c21167280a4d669123bb1ed7675d79, State: Initialized, Role: LEARNER
I20250622 13:53:01.946489 4379 consensus_queue.cc:260] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:01.947438 4379 ts_tablet_manager.cc:1428] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:53:01.948092 3844 tablet_copy_service.cc:342] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Request end of tablet copy session 52c21167280a4d669123bb1ed7675d79-1aa8fa3d766949b99d0538f8b76498d5 received from {username='slave'} at 127.0.0.1:45650
I20250622 13:53:01.948202 3844 tablet_copy_service.cc:434] P 946d416d6f4a4b0ea6a77b93cd6f58f6: ending tablet copy session 52c21167280a4d669123bb1ed7675d79-1aa8fa3d766949b99d0538f8b76498d5 on tablet 1aa8fa3d766949b99d0538f8b76498d5 with peer 52c21167280a4d669123bb1ed7675d79
I20250622 13:53:01.952697 4384 tablet_copy_client.cc:538] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:01.956319 4384 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: Bootstrap starting.
I20250622 13:53:01.963560 4378 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:01.964030 4378 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: Bootstrap complete.
I20250622 13:53:01.964308 4378 ts_tablet_manager.cc:1397] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: Time spent bootstrapping tablet: real 0.049s user 0.043s sys 0.004s
I20250622 13:53:01.965068 4378 raft_consensus.cc:357] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:01.965279 4378 raft_consensus.cc:738] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 52c21167280a4d669123bb1ed7675d79, State: Initialized, Role: LEARNER
I20250622 13:53:01.965529 4378 consensus_queue.cc:260] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:01.966218 4378 ts_tablet_manager.cc:1428] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250622 13:53:01.966948 3918 tablet_copy_service.cc:342] P 3377fd0584c546c1a938e3a1204b2be3: Request end of tablet copy session 52c21167280a4d669123bb1ed7675d79-91736fef6df543c4953497186dfe1c5e received from {username='slave'} at 127.0.0.1:36376
I20250622 13:53:01.967098 3918 tablet_copy_service.cc:434] P 3377fd0584c546c1a938e3a1204b2be3: ending tablet copy session 52c21167280a4d669123bb1ed7675d79-91736fef6df543c4953497186dfe1c5e on tablet 91736fef6df543c4953497186dfe1c5e with peer 52c21167280a4d669123bb1ed7675d79
I20250622 13:53:01.988739 4386 tablet_copy_client.cc:538] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:01.991770 4386 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap starting.
I20250622 13:53:01.995385 4384 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:01.995762 4384 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: Bootstrap complete.
I20250622 13:53:01.996048 4384 ts_tablet_manager.cc:1397] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.009s
I20250622 13:53:01.996748 4384 raft_consensus.cc:357] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:01.996970 4384 raft_consensus.cc:738] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d23b47356efa46c792ab125b568ea95a, State: Initialized, Role: LEARNER
I20250622 13:53:01.997154 4384 consensus_queue.cc:260] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:01.997634 4384 ts_tablet_manager.cc:1428] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250622 13:53:01.998358 3769 tablet_copy_service.cc:342] P d13e1e991fdc47899aac5b7c33fbbbd9: Request end of tablet copy session d23b47356efa46c792ab125b568ea95a-f10daa2b314a4660bd36fa7902789a8e received from {username='slave'} at 127.0.0.1:53832
I20250622 13:53:01.998466 3769 tablet_copy_service.cc:434] P d13e1e991fdc47899aac5b7c33fbbbd9: ending tablet copy session d23b47356efa46c792ab125b568ea95a-f10daa2b314a4660bd36fa7902789a8e on tablet f10daa2b314a4660bd36fa7902789a8e with peer d23b47356efa46c792ab125b568ea95a
I20250622 13:53:02.029784 4386 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:02.030165 4386 tablet_bootstrap.cc:492] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap complete.
I20250622 13:53:02.030418 4386 ts_tablet_manager.cc:1397] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: Time spent bootstrapping tablet: real 0.039s user 0.039s sys 0.000s
I20250622 13:53:02.031083 4386 raft_consensus.cc:357] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:02.031303 4386 raft_consensus.cc:738] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: feaebd1ce3104a2fa3cc207fd3c23468, State: Initialized, Role: LEARNER
I20250622 13:53:02.031486 4386 consensus_queue.cc:260] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:02.032136 4386 ts_tablet_manager.cc:1428] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:02.032821 3918 tablet_copy_service.cc:342] P 3377fd0584c546c1a938e3a1204b2be3: Request end of tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-d317ca8305314d45b47fa1014f3b5162 received from {username='slave'} at 127.0.0.1:36388
I20250622 13:53:02.032932 3918 tablet_copy_service.cc:434] P 3377fd0584c546c1a938e3a1204b2be3: ending tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-d317ca8305314d45b47fa1014f3b5162 on tablet d317ca8305314d45b47fa1014f3b5162 with peer feaebd1ce3104a2fa3cc207fd3c23468
I20250622 13:53:02.035885 4389 tablet_copy_client.cc:538] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:02.038882 4389 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap starting.
I20250622 13:53:02.074541 4389 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=99 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:02.074908 4389 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap complete.
I20250622 13:53:02.075150 4389 ts_tablet_manager.cc:1397] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: Time spent bootstrapping tablet: real 0.036s user 0.034s sys 0.001s
I20250622 13:53:02.075821 4389 raft_consensus.cc:357] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:02.075969 4389 raft_consensus.cc:738] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: feaebd1ce3104a2fa3cc207fd3c23468, State: Initialized, Role: LEARNER
I20250622 13:53:02.076145 4389 consensus_queue.cc:260] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:02.076566 4389 ts_tablet_manager.cc:1428] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250622 13:53:02.077695 3918 tablet_copy_service.cc:342] P 3377fd0584c546c1a938e3a1204b2be3: Request end of tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-677dddfcdf864e7f9b997d196b58bc78 received from {username='slave'} at 127.0.0.1:36388
I20250622 13:53:02.077824 3918 tablet_copy_service.cc:434] P 3377fd0584c546c1a938e3a1204b2be3: ending tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-677dddfcdf864e7f9b997d196b58bc78 on tablet 677dddfcdf864e7f9b997d196b58bc78 with peer feaebd1ce3104a2fa3cc207fd3c23468
I20250622 13:53:02.134944 4184 raft_consensus.cc:1215] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250622 13:53:02.152935 4259 raft_consensus.cc:1215] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250622 13:53:02.202961 4259 raft_consensus.cc:1215] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250622 13:53:02.261225 4184 raft_consensus.cc:1215] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250622 13:53:02.288167 4333 raft_consensus.cc:1215] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250622 13:53:02.290036 4333 raft_consensus.cc:1215] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250622 13:53:02.565116 4391 raft_consensus.cc:1062] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: attempting to promote NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 to VOTER
I20250622 13:53:02.566057 4391 consensus_queue.cc:237] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:02.568797 4259 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.568849 3828 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.569494 3753 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.569245 4410 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.569669 4391 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.569931 4413 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.573704 4413 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.574174 4259 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.577575 3828 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.578413 3651 catalog_manager.cc:5582] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:02.578330 3753 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.588920 3903 consensus_queue.cc:237] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:02.590648 3753 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.590631 4259 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.591147 4412 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.591374 4391 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.593456 4410 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.594146 4259 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.594779 3753 raft_consensus.cc:2953] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.598004 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d317ca8305314d45b47fa1014f3b5162 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:02.599644 3656 catalog_manager.cc:5582] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:02.603099 3819 tablet_service.cc:1515] Processing DeleteTablet for tablet d317ca8305314d45b47fa1014f3b5162 with delete_type TABLET_DATA_TOMBSTONED (TS 946d416d6f4a4b0ea6a77b93cd6f58f6 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:45604
I20250622 13:53:02.603950 4421 tablet_replica.cc:331] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:02.604157 4421 raft_consensus.cc:2241] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:02.604349 4421 raft_consensus.cc:2270] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:02.605185 4421 ts_tablet_manager.cc:1905] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:02.610656 4421 ts_tablet_manager.cc:1918] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250622 13:53:02.610783 4421 log.cc:1199] T d317ca8305314d45b47fa1014f3b5162 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root/wals/d317ca8305314d45b47fa1014f3b5162
I20250622 13:53:02.611402 3643 catalog_manager.cc:4928] TS 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): tablet d317ca8305314d45b47fa1014f3b5162 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:02.744594 4423 raft_consensus.cc:1062] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: attempting to promote NON_VOTER d23b47356efa46c792ab125b568ea95a to VOTER
I20250622 13:53:02.745338 4423 consensus_queue.cc:237] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:02.747726 4184 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.747910 3828 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.748086 3903 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.748574 4119 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.749114 4119 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.749577 4119 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.754094 4422 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.754746 4184 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.759624 3903 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.764704 3828 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.768237 3652 catalog_manager.cc:5582] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.771544 4432 raft_consensus.cc:1062] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: attempting to promote NON_VOTER 52c21167280a4d669123bb1ed7675d79 to VOTER
I20250622 13:53:02.772761 4432 consensus_queue.cc:237] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:02.774474 3753 consensus_queue.cc:237] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:02.776448 3753 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.776906 4431 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.778002 4184 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.778183 3908 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.778328 4333 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 LEARNER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.778622 4422 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.778865 4431 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.780493 3828 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.781224 4422 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.781416 4409 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.782452 4429 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.784941 3828 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.786142 4184 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.786989 4410 raft_consensus.cc:1062] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: attempting to promote NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 to VOTER
I20250622 13:53:02.787775 4410 consensus_queue.cc:237] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:02.791134 4259 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.791352 3828 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.791802 3644 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f10daa2b314a4660bd36fa7902789a8e with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:02.791896 3656 catalog_manager.cc:5582] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 29 to 30, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.792546 4413 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.002s
I20250622 13:53:02.794513 4391 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.792763 3753 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.795544 4432 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.796505 4391 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.797727 3890 tablet_service.cc:1515] Processing DeleteTablet for tablet f10daa2b314a4660bd36fa7902789a8e with delete_type TABLET_DATA_TOMBSTONED (TS 3377fd0584c546c1a938e3a1204b2be3 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:36332
I20250622 13:53:02.803268 3908 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.813040 3651 catalog_manager.cc:5582] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.817595 4409 raft_consensus.cc:1062] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: attempting to promote NON_VOTER d23b47356efa46c792ab125b568ea95a to VOTER
I20250622 13:53:02.818495 4409 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:02.818763 4333 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.823153 3908 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.824572 4441 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.825232 4184 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.826503 4440 tablet_replica.cc:331] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:02.827632 4431 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.828806 4440 raft_consensus.cc:2241] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:02.829715 4440 raft_consensus.cc:2270] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:02.831753 4440 ts_tablet_manager.cc:1905] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:02.845784 4440 ts_tablet_manager.cc:1918] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250622 13:53:02.846051 4440 log.cc:1199] T f10daa2b314a4660bd36fa7902789a8e P 3377fd0584c546c1a938e3a1204b2be3: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root/wals/f10daa2b314a4660bd36fa7902789a8e
I20250622 13:53:02.847082 3643 catalog_manager.cc:4928] TS 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): tablet f10daa2b314a4660bd36fa7902789a8e (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:02.850250 4441 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.855557 3753 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.855165 4413 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.855971 3756 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.857900 3908 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.859126 4183 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.860983 4441 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.862196 4259 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.864061 3828 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.863196 3651 catalog_manager.cc:5582] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 28 to 29, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:02.868744 3753 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.871366 3903 consensus_queue.cc:237] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:02.873078 4258 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.869231 3655 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:02.875742 3829 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.876745 3756 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.877280 4391 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.877749 3829 consensus_queue.cc:237] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:02.878135 4391 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.879769 4333 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.879971 4410 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.880805 3753 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.880954 3832 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.882210 4259 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.880309 4443 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.003s
I20250622 13:53:02.882993 4441 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.885496 4413 raft_consensus.cc:1062] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: attempting to promote NON_VOTER 52c21167280a4d669123bb1ed7675d79 to VOTER
I20250622 13:53:02.885936 3828 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:02.886266 4413 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:02.888792 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 677dddfcdf864e7f9b997d196b58bc78 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:02.888846 4183 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.889809 3652 catalog_manager.cc:5582] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index 29 to 30, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:02.891283 3908 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.889174 4333 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.887305 4441 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.891829 4445 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.895129 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 LEARNER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.895525 3755 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.895998 4412 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.896520 4412 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.897548 3828 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:02.898908 3756 raft_consensus.cc:2953] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.900038 4412 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:02.904098 3745 tablet_service.cc:1515] Processing DeleteTablet for tablet 677dddfcdf864e7f9b997d196b58bc78 with delete_type TABLET_DATA_TOMBSTONED (TS d13e1e991fdc47899aac5b7c33fbbbd9 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:53772
I20250622 13:53:02.904392 3652 catalog_manager.cc:5582] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 reported cstate change: config changed from index 29 to 30, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.910259 3890 tablet_service.cc:1515] Processing DeleteTablet for tablet 1aa8fa3d766949b99d0538f8b76498d5 with delete_type TABLET_DATA_TOMBSTONED (TS 3377fd0584c546c1a938e3a1204b2be3 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:36332
I20250622 13:53:02.910620 4440 tablet_replica.cc:331] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:02.910894 4440 raft_consensus.cc:2241] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:02.911128 4440 raft_consensus.cc:2270] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:02.912444 4440 ts_tablet_manager.cc:1905] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:02.913477 4443 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.916469 4445 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.919924 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c9260a075f104769bc81ca7eb0c76cb2 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:02.920820 3908 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.921452 4413 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.922379 4183 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.926918 4454 tablet_replica.cc:331] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:02.927213 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1aa8fa3d766949b99d0538f8b76498d5 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:02.927472 3753 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.927981 4440 ts_tablet_manager.cc:1918] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250622 13:53:02.928550 4440 log.cc:1199] T 1aa8fa3d766949b99d0538f8b76498d5 P 3377fd0584c546c1a938e3a1204b2be3: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root/wals/1aa8fa3d766949b99d0538f8b76498d5
I20250622 13:53:02.927798 3651 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 29 to 30, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:02.930824 4332 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.932359 3745 tablet_service.cc:1515] Processing DeleteTablet for tablet c9260a075f104769bc81ca7eb0c76cb2 with delete_type TABLET_DATA_TOMBSTONED (TS d13e1e991fdc47899aac5b7c33fbbbd9 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:53772
I20250622 13:53:02.934176 3832 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.927222 4454 raft_consensus.cc:2241] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:02.935240 4454 raft_consensus.cc:2270] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:02.936120 4454 ts_tablet_manager.cc:1905] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:02.936655 3656 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 28 to 29, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:02.939728 3643 catalog_manager.cc:4928] TS 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): tablet 1aa8fa3d766949b99d0538f8b76498d5 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:02.942689 4454 ts_tablet_manager.cc:1918] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250622 13:53:02.942833 4454 log.cc:1199] T 677dddfcdf864e7f9b997d196b58bc78 P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root/wals/677dddfcdf864e7f9b997d196b58bc78
I20250622 13:53:02.944111 3908 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:02.945250 4454 tablet_replica.cc:331] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:02.945317 3644 catalog_manager.cc:4928] TS d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517): tablet 677dddfcdf864e7f9b997d196b58bc78 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:02.945566 4454 raft_consensus.cc:2241] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:02.945725 3753 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.945827 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:02.945783 4454 raft_consensus.cc:2270] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:02.946280 4410 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:02.947175 4454 ts_tablet_manager.cc:1905] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:02.946611 4413 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20250622 13:53:02.948385 4410 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.950508 4332 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.952244 4454 ts_tablet_manager.cc:1918] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250622 13:53:02.952389 4454 log.cc:1199] T c9260a075f104769bc81ca7eb0c76cb2 P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root/wals/c9260a075f104769bc81ca7eb0c76cb2
I20250622 13:53:02.953097 3644 catalog_manager.cc:4928] TS d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517): tablet c9260a075f104769bc81ca7eb0c76cb2 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:02.953438 3753 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:02.953717 3652 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:02.954711 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 91736fef6df543c4953497186dfe1c5e with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:02.957404 3819 tablet_service.cc:1515] Processing DeleteTablet for tablet 91736fef6df543c4953497186dfe1c5e with delete_type TABLET_DATA_TOMBSTONED (TS 946d416d6f4a4b0ea6a77b93cd6f58f6 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:45604
I20250622 13:53:02.957705 4421 tablet_replica.cc:331] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:02.957880 4421 raft_consensus.cc:2241] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:02.958040 4421 raft_consensus.cc:2270] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:02.958817 4421 ts_tablet_manager.cc:1905] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:02.963341 4421 ts_tablet_manager.cc:1918] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250622 13:53:02.963424 4421 log.cc:1199] T 91736fef6df543c4953497186dfe1c5e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root/wals/91736fef6df543c4953497186dfe1c5e
I20250622 13:53:02.963948 3643 catalog_manager.cc:4928] TS 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): tablet 91736fef6df543c4953497186dfe1c5e (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:03.712708 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:03.720029 3832 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1aa8fa3d766949b99d0538f8b76498d5"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "52c21167280a4d669123bb1ed7675d79"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:03.720232 3832 raft_consensus.cc:604] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Received request to transfer leadership to TS 52c21167280a4d669123bb1ed7675d79
I20250622 13:53:03.721204 3903 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "677dddfcdf864e7f9b997d196b58bc78"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:53:03.721378 3903 raft_consensus.cc:604] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Received request to transfer leadership to TS feaebd1ce3104a2fa3cc207fd3c23468
I20250622 13:53:03.730022 3753 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:03.730192 3753 raft_consensus.cc:604] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Received request to transfer leadership to TS 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:53:03.731072 3753 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:03.731206 3753 raft_consensus.cc:604] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Received request to transfer leadership to TS 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:53:03.731974 3832 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:03.732111 3832 raft_consensus.cc:604] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Received request to transfer leadership to TS 3377fd0584c546c1a938e3a1204b2be3
I20250622 13:53:03.733039 3753 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "d23b47356efa46c792ab125b568ea95a"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:03.733153 3753 raft_consensus.cc:604] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Received request to transfer leadership to TS d23b47356efa46c792ab125b568ea95a
I20250622 13:53:03.733489 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250622 13:53:03.733667 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:03.741786 4413 raft_consensus.cc:991] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower feaebd1ce3104a2fa3cc207fd3c23468 to start an election
I20250622 13:53:03.741921 4413 raft_consensus.cc:1079] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Signalling peer feaebd1ce3104a2fa3cc207fd3c23468 to start an election
I20250622 13:53:03.742457 4259 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "677dddfcdf864e7f9b997d196b58bc78"
dest_uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
from {username='slave'} at 127.0.0.1:33372
I20250622 13:53:03.742617 4259 raft_consensus.cc:491] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:03.742707 4259 raft_consensus.cc:3058] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:03.744551 4259 raft_consensus.cc:513] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:03.745209 4259 leader_election.cc:290] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [CANDIDATE]: Term 2 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:03.746174 3903 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "677dddfcdf864e7f9b997d196b58bc78" candidate_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:03.746308 3903 raft_consensus.cc:3053] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:03.746372 3903 raft_consensus.cc:738] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:03.746644 3903 consensus_queue.cc:260] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:03.746927 3903 raft_consensus.cc:3058] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:03.748883 3903 raft_consensus.cc:2466] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feaebd1ce3104a2fa3cc207fd3c23468 in term 2.
I20250622 13:53:03.749280 4224 leader_election.cc:304] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, feaebd1ce3104a2fa3cc207fd3c23468; no voters:
I20250622 13:53:03.749964 4460 raft_consensus.cc:2802] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:03.750468 4460 raft_consensus.cc:695] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Becoming Leader. State: Replica: feaebd1ce3104a2fa3cc207fd3c23468, State: Running, Role: LEADER
I20250622 13:53:03.750658 3832 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "677dddfcdf864e7f9b997d196b58bc78" candidate_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:03.750793 3832 raft_consensus.cc:3058] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:03.750749 4460 consensus_queue.cc:237] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:03.752759 3832 raft_consensus.cc:2466] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feaebd1ce3104a2fa3cc207fd3c23468 in term 2.
I20250622 13:53:03.754328 3656 catalog_manager.cc:5582] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 reported cstate change: term changed from 1 to 2, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69). New cstate: current_term: 2 leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:03.783176 4422 raft_consensus.cc:991] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:03.783385 4422 raft_consensus.cc:1079] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Signalling peer 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:03.783926 3832 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:45606
I20250622 13:53:03.784087 3832 raft_consensus.cc:491] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:03.784176 3832 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:03.786051 3832 raft_consensus.cc:513] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:03.786572 3832 leader_election.cc:290] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 3 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:03.786957 3903 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:03.786957 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:03.787175 3753 raft_consensus.cc:3053] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:53:03.787176 3903 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:03.787269 3753 raft_consensus.cc:738] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:03.787514 3753 consensus_queue.cc:260] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:03.787796 3753 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:03.789918 3903 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 3.
I20250622 13:53:03.790249 3753 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 3.
I20250622 13:53:03.790643 3800 leader_election.cc:304] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:03.790903 4432 raft_consensus.cc:2802] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:53:03.791219 4432 raft_consensus.cc:695] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:03.791468 4432 consensus_queue.cc:237] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:03.793774 3656 catalog_manager.cc:5582] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 2 to 3, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 3 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:03.946442 4466 raft_consensus.cc:991] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower 52c21167280a4d669123bb1ed7675d79 to start an election
I20250622 13:53:03.946610 4466 raft_consensus.cc:1079] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Signalling peer 52c21167280a4d669123bb1ed7675d79 to start an election
I20250622 13:53:03.947145 4332 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1aa8fa3d766949b99d0538f8b76498d5"
dest_uuid: "52c21167280a4d669123bb1ed7675d79"
from {username='slave'} at 127.0.0.1:35746
I20250622 13:53:03.947340 4332 raft_consensus.cc:491] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:03.947423 4332 raft_consensus.cc:3058] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:03.949213 4332 raft_consensus.cc:513] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:03.949905 4332 leader_election.cc:290] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [CANDIDATE]: Term 2 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517)
I20250622 13:53:03.950215 3832 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8fa3d766949b99d0538f8b76498d5" candidate_uuid: "52c21167280a4d669123bb1ed7675d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:03.950470 3832 raft_consensus.cc:3053] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:03.950598 3832 raft_consensus.cc:738] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:03.950879 3832 consensus_queue.cc:260] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:03.951174 3832 raft_consensus.cc:3058] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:03.953794 3832 raft_consensus.cc:2466] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c21167280a4d669123bb1ed7675d79 in term 2.
I20250622 13:53:03.954154 4298 leader_election.cc:304] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c21167280a4d669123bb1ed7675d79, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:53:03.956514 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa8fa3d766949b99d0538f8b76498d5" candidate_uuid: "52c21167280a4d669123bb1ed7675d79" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:03.956576 4469 raft_consensus.cc:2802] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:03.956686 3753 raft_consensus.cc:3058] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:03.959501 3753 raft_consensus.cc:2466] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c21167280a4d669123bb1ed7675d79 in term 2.
I20250622 13:53:03.960000 4469 raft_consensus.cc:695] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 2 LEADER]: Becoming Leader. State: Replica: 52c21167280a4d669123bb1ed7675d79, State: Running, Role: LEADER
I20250622 13:53:03.960309 4469 consensus_queue.cc:237] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:03.963243 3652 catalog_manager.cc:5582] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 reported cstate change: term changed from 1 to 2, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to 52c21167280a4d669123bb1ed7675d79 (127.0.103.70). New cstate: current_term: 2 leader_uuid: "52c21167280a4d669123bb1ed7675d79" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:03.963308 3756 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:03.965425 3832 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20250622 13:53:03.965778 3722 consensus_peers.cc:487] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 -> Peer d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811): Couldn't send request to peer d23b47356efa46c792ab125b568ea95a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5246e7fae8d44ba6a5e3ac826f9988f4. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:03.966022 4422 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:03.966490 3903 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:53:03.966940 4422 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:03.969230 4465 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:03.975189 3832 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:03.978997 3652 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:03.979223 3903 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:03.979908 3908 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:03.982410 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
W20250622 13:53:03.982494 3874 consensus_peers.cc:487] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 -> Peer feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925): Couldn't send request to peer feaebd1ce3104a2fa3cc207fd3c23468. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91736fef6df543c4953497186dfe1c5e. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:03.982848 3756 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250622 13:53:03.983358 4458 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:03.983659 4413 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:03.987555 4413 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:03.988206 4332 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:03.989542 3756 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
W20250622 13:53:03.993867 3705 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20250622 13:53:03.993548 3652 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 30 to 31, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250622 13:53:03.998222 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b045aeb5889a4b88b8ee72370f2a7e39, TS 3377fd0584c546c1a938e3a1204b2be3 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:03.998308 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b045aeb5889a4b88b8ee72370f2a7e39, TS 3377fd0584c546c1a938e3a1204b2be3 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:04.001610 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59, TS d13e1e991fdc47899aac5b7c33fbbbd9 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:04.001678 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59, TS d13e1e991fdc47899aac5b7c33fbbbd9 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:04.004618 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6fd3fa41ece64410b7448f59d5db9ba7, TS 3377fd0584c546c1a938e3a1204b2be3 -> TS feaebd1ce3104a2fa3cc207fd3c23468 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:04.004683 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6fd3fa41ece64410b7448f59d5db9ba7, TS 3377fd0584c546c1a938e3a1204b2be3 -> TS feaebd1ce3104a2fa3cc207fd3c23468 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:04.022799 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 677dddfcdf864e7f9b997d196b58bc78, TS 946d416d6f4a4b0ea6a77b93cd6f58f6 -> TS d23b47356efa46c792ab125b568ea95a move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
W20250622 13:53:04.022892 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 677dddfcdf864e7f9b997d196b58bc78, TS 946d416d6f4a4b0ea6a77b93cd6f58f6 -> TS d23b47356efa46c792ab125b568ea95a move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:04.068540 4422 raft_consensus.cc:991] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower d23b47356efa46c792ab125b568ea95a to start an election
I20250622 13:53:04.068693 4422 raft_consensus.cc:1079] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Signalling peer d23b47356efa46c792ab125b568ea95a to start an election
I20250622 13:53:04.069278 4183 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e"
dest_uuid: "d23b47356efa46c792ab125b568ea95a"
from {username='slave'} at 127.0.0.1:38558
I20250622 13:53:04.069499 4183 raft_consensus.cc:491] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:04.069597 4183 raft_consensus.cc:3058] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:04.071926 4183 raft_consensus.cc:513] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:04.072830 3756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "d23b47356efa46c792ab125b568ea95a" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:04.072983 3832 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10daa2b314a4660bd36fa7902789a8e" candidate_uuid: "d23b47356efa46c792ab125b568ea95a" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:04.073072 3756 raft_consensus.cc:3053] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:04.073100 3832 raft_consensus.cc:3058] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:04.073137 3756 raft_consensus.cc:738] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:04.073170 4183 leader_election.cc:290] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [CANDIDATE]: Term 2 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:53:04.073331 3756 consensus_queue.cc:260] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:04.073665 3756 raft_consensus.cc:3058] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:04.076102 3832 raft_consensus.cc:2466] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23b47356efa46c792ab125b568ea95a in term 2.
I20250622 13:53:04.076340 3756 raft_consensus.cc:2466] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23b47356efa46c792ab125b568ea95a in term 2.
I20250622 13:53:04.076500 4149 leader_election.cc:304] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d23b47356efa46c792ab125b568ea95a; no voters:
I20250622 13:53:04.077656 4422 raft_consensus.cc:991] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:04.077754 4482 raft_consensus.cc:2802] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:04.077788 4465 raft_consensus.cc:1079] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Signalling peer 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:04.078380 4482 raft_consensus.cc:695] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 2 LEADER]: Becoming Leader. State: Replica: d23b47356efa46c792ab125b568ea95a, State: Running, Role: LEADER
I20250622 13:53:04.078459 3832 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:45606
I20250622 13:53:04.078641 3832 raft_consensus.cc:491] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:04.078727 3832 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:04.078717 4482 consensus_queue.cc:237] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:04.082824 3832 raft_consensus.cc:513] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:04.083611 3832 leader_election.cc:290] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 3 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:04.083959 3756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:04.084175 3756 raft_consensus.cc:3053] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:53:04.084134 3903 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:04.084275 3756 raft_consensus.cc:738] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:04.084335 3903 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:04.084609 3756 consensus_queue.cc:260] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:04.084905 3756 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:04.086688 3903 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 3.
I20250622 13:53:04.087026 3799 leader_election.cc:304] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:53:04.087230 3756 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 3.
I20250622 13:53:04.087255 4466 raft_consensus.cc:2802] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:53:04.087402 4466 raft_consensus.cc:695] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:04.087620 4466 consensus_queue.cc:237] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:04.088706 3656 catalog_manager.cc:5582] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a reported cstate change: term changed from 1 to 2, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to d23b47356efa46c792ab125b568ea95a (127.0.103.68). New cstate: current_term: 2 leader_uuid: "d23b47356efa46c792ab125b568ea95a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:04.090529 3651 catalog_manager.cc:5582] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 2 to 3, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 3 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:04.131454 4432 raft_consensus.cc:991] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:04.131616 4432 raft_consensus.cc:1079] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Signalling peer 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:04.132128 3908 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:36344
I20250622 13:53:04.132326 3908 raft_consensus.cc:491] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:04.132427 3908 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:04.134831 3908 raft_consensus.cc:513] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:04.135401 3908 leader_election.cc:290] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 2 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:53:04.135851 3832 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:04.135851 3756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:04.136067 3832 raft_consensus.cc:3053] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:04.136075 3756 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:04.136184 3832 raft_consensus.cc:738] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:04.136454 3832 consensus_queue.cc:260] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:04.136791 3832 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:04.139518 3832 raft_consensus.cc:2466] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 2.
I20250622 13:53:04.139518 3756 raft_consensus.cc:2466] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 2.
I20250622 13:53:04.139932 3874 leader_election.cc:304] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:04.140216 4477 raft_consensus.cc:2802] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:04.140365 4477 raft_consensus.cc:695] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:04.140585 4477 consensus_queue.cc:237] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:04.143168 3656 catalog_manager.cc:5582] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 1 to 2, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250622 13:53:04.329303 3908 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:04.329855 4487 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:04.335081 3832 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:04.336328 4489 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:04.376765 3908 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250622 13:53:04.377277 4432 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:04.378417 3756 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250622 13:53:04.385699 4496 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.002s
I20250622 13:53:04.391238 4499 ts_tablet_manager.cc:927] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: Initiating tablet copy from peer d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517)
I20250622 13:53:04.391985 4499 tablet_copy_client.cc:323] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:40517
I20250622 13:53:04.392542 3769 tablet_copy_service.cc:140] P d13e1e991fdc47899aac5b7c33fbbbd9: Received BeginTabletCopySession request for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 from peer d23b47356efa46c792ab125b568ea95a ({username='slave'} at 127.0.0.1:53832)
I20250622 13:53:04.392679 3769 tablet_copy_service.cc:161] P d13e1e991fdc47899aac5b7c33fbbbd9: Beginning new tablet copy session on tablet 5246e7fae8d44ba6a5e3ac826f9988f4 from peer d23b47356efa46c792ab125b568ea95a at {username='slave'} at 127.0.0.1:53832: session id = d23b47356efa46c792ab125b568ea95a-5246e7fae8d44ba6a5e3ac826f9988f4
I20250622 13:53:04.394879 3769 tablet_copy_source_session.cc:215] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:04.396085 4499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5246e7fae8d44ba6a5e3ac826f9988f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:04.401571 4499 tablet_copy_client.cc:806] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:04.401851 4499 tablet_copy_client.cc:670] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:04.403087 4500 tablet_copy_client.cc:1025] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:04.431851 4501 ts_tablet_manager.cc:927] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: Initiating tablet copy from peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:04.432439 4501 tablet_copy_client.cc:323] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33649
I20250622 13:53:04.432997 3918 tablet_copy_service.cc:140] P 3377fd0584c546c1a938e3a1204b2be3: Received BeginTabletCopySession request for tablet 91736fef6df543c4953497186dfe1c5e from peer feaebd1ce3104a2fa3cc207fd3c23468 ({username='slave'} at 127.0.0.1:36388)
I20250622 13:53:04.433125 3918 tablet_copy_service.cc:161] P 3377fd0584c546c1a938e3a1204b2be3: Beginning new tablet copy session on tablet 91736fef6df543c4953497186dfe1c5e from peer feaebd1ce3104a2fa3cc207fd3c23468 at {username='slave'} at 127.0.0.1:36388: session id = feaebd1ce3104a2fa3cc207fd3c23468-91736fef6df543c4953497186dfe1c5e
I20250622 13:53:04.435271 3918 tablet_copy_source_session.cc:215] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:04.436355 4501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91736fef6df543c4953497186dfe1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:04.441527 4501 tablet_copy_client.cc:806] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:04.441781 4501 tablet_copy_client.cc:670] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:04.443336 4502 tablet_copy_client.cc:1025] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:04.467999 3832 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer 52c21167280a4d669123bb1ed7675d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:04.468559 4469 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:04.472143 3753 raft_consensus.cc:1273] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Refusing update from remote peer 52c21167280a4d669123bb1ed7675d79: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:04.472645 4469 consensus_queue.cc:1035] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:04.560698 3753 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250622 13:53:04.561230 4496 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:04.564438 3908 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250622 13:53:04.565032 4432 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:04.595669 3832 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:04.596215 4509 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:04.600741 3753 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:04.601258 4509 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:04.654592 3753 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250622 13:53:04.655120 4477 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:04.659216 3832 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250622 13:53:04.659729 4477 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:05.733959 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:05.736989 3828 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:05.737159 3828 raft_consensus.cc:604] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:53:05.738169 3903 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:53:05.738315 3903 raft_consensus.cc:604] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:53:05.739235 3828 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:05.739395 3828 raft_consensus.cc:604] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Received request to transfer leadership to TS 3377fd0584c546c1a938e3a1204b2be3
I20250622 13:53:05.739696 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250622 13:53:05.739903 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:05.808130 4516 raft_consensus.cc:991] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:05.808290 4516 raft_consensus.cc:1079] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Signalling peer d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:05.808912 3756 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:53794
I20250622 13:53:05.809087 3756 raft_consensus.cc:491] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:05.809156 3756 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:05.811446 3756 raft_consensus.cc:513] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:05.811985 3756 leader_election.cc:290] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 3 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:05.812279 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:05.812435 3903 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bac2340a96e94c219f59c12945eb8bf6" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:05.812449 3828 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:05.812616 3903 raft_consensus.cc:3053] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:53:05.812687 3903 raft_consensus.cc:738] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:05.812969 3903 consensus_queue.cc:260] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:05.813231 3903 raft_consensus.cc:3058] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:05.814671 3828 raft_consensus.cc:2466] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 3.
I20250622 13:53:05.815080 3724 leader_election.cc:304] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:05.815361 4524 raft_consensus.cc:2802] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:53:05.815496 4524 raft_consensus.cc:695] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:05.815703 4524 consensus_queue.cc:237] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:05.816226 3903 raft_consensus.cc:2466] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 3.
I20250622 13:53:05.818244 3656 catalog_manager.cc:5582] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 2 to 3, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 3 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:05.984351 4530 raft_consensus.cc:991] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:05.984514 4530 raft_consensus.cc:1079] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Signalling peer 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:05.985117 3903 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:36344
I20250622 13:53:05.985415 3903 raft_consensus.cc:491] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:05.985556 3903 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:05.987725 3903 raft_consensus.cc:513] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:05.988440 3903 leader_election.cc:290] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 2 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
I20250622 13:53:05.988729 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:05.988921 3828 raft_consensus.cc:3053] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:05.989017 3828 raft_consensus.cc:738] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:05.989248 3828 consensus_queue.cc:260] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:05.989681 3828 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:05.991595 3828 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 2.
I20250622 13:53:05.991982 3873 leader_election.cc:304] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:53:05.992233 4520 raft_consensus.cc:2802] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:05.992374 4520 raft_consensus.cc:695] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:05.992628 4520 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:05.998426 4183 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "d23b47356efa46c792ab125b568ea95a"
I20250622 13:53:05.998636 4183 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:06.000393 4183 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 2.
I20250622 13:53:06.001540 3656 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 1 to 2, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:06.163512 4432 raft_consensus.cc:991] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:06.163677 4432 raft_consensus.cc:1079] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Signalling peer d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:06.164198 3753 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:53780
I20250622 13:53:06.164412 3753 raft_consensus.cc:491] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:06.164521 3753 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:06.167236 3753 raft_consensus.cc:513] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:06.167855 3753 leader_election.cc:290] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 4 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:06.168231 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:06.168255 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:06.168434 3828 raft_consensus.cc:3053] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Stepping down as leader of term 3
I20250622 13:53:06.168435 3908 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:06.168540 3828 raft_consensus.cc:738] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:06.168767 3828 consensus_queue.cc:260] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:06.169057 3828 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:06.171406 3908 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 4.
I20250622 13:53:06.171602 3828 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 4.
I20250622 13:53:06.171849 3724 leader_election.cc:304] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:06.172091 4524 raft_consensus.cc:2802] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 13:53:06.172237 4524 raft_consensus.cc:695] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:06.172482 4524 consensus_queue.cc:237] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:06.175215 3656 catalog_manager.cc:5582] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 3 to 4, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 4 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:06.360383 3828 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250622 13:53:06.360883 4524 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:06.366989 3903 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250622 13:53:06.367556 4524 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:06.403445 4499 tablet_copy_client.cc:538] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:06.406296 4499 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: Bootstrap starting.
I20250622 13:53:06.443581 4499 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:06.443650 4501 tablet_copy_client.cc:538] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:06.444020 4499 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: Bootstrap complete.
I20250622 13:53:06.444203 4499 ts_tablet_manager.cc:1397] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: Time spent bootstrapping tablet: real 0.038s user 0.038s sys 0.000s
I20250622 13:53:06.444804 4499 raft_consensus.cc:357] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:06.444931 4499 raft_consensus.cc:738] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d23b47356efa46c792ab125b568ea95a, State: Initialized, Role: LEARNER
I20250622 13:53:06.445039 4499 consensus_queue.cc:260] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:06.445513 4499 ts_tablet_manager.cc:1428] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250622 13:53:06.446297 4501 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap starting.
I20250622 13:53:06.448163 3769 tablet_copy_service.cc:342] P d13e1e991fdc47899aac5b7c33fbbbd9: Request end of tablet copy session d23b47356efa46c792ab125b568ea95a-5246e7fae8d44ba6a5e3ac826f9988f4 received from {username='slave'} at 127.0.0.1:53832
I20250622 13:53:06.448318 3769 tablet_copy_service.cc:434] P d13e1e991fdc47899aac5b7c33fbbbd9: ending tablet copy session d23b47356efa46c792ab125b568ea95a-5246e7fae8d44ba6a5e3ac826f9988f4 on tablet 5246e7fae8d44ba6a5e3ac826f9988f4 with peer d23b47356efa46c792ab125b568ea95a
I20250622 13:53:06.452613 4183 raft_consensus.cc:1215] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250622 13:53:06.488390 4501 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:06.488826 4501 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap complete.
I20250622 13:53:06.489080 4501 ts_tablet_manager.cc:1397] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: Time spent bootstrapping tablet: real 0.043s user 0.039s sys 0.000s
I20250622 13:53:06.489866 4501 raft_consensus.cc:357] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:06.490072 4501 raft_consensus.cc:738] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: feaebd1ce3104a2fa3cc207fd3c23468, State: Initialized, Role: LEARNER
I20250622 13:53:06.490310 4501 consensus_queue.cc:260] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:06.490788 4501 ts_tablet_manager.cc:1428] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250622 13:53:06.491348 3918 tablet_copy_service.cc:342] P 3377fd0584c546c1a938e3a1204b2be3: Request end of tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-91736fef6df543c4953497186dfe1c5e received from {username='slave'} at 127.0.0.1:36388
I20250622 13:53:06.491474 3918 tablet_copy_service.cc:434] P 3377fd0584c546c1a938e3a1204b2be3: ending tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-91736fef6df543c4953497186dfe1c5e on tablet 91736fef6df543c4953497186dfe1c5e with peer feaebd1ce3104a2fa3cc207fd3c23468
I20250622 13:53:06.501142 4259 raft_consensus.cc:1215] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.30->[1.31-1.31] Dedup: 1.31->[]
I20250622 13:53:06.560010 3828 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:06.560479 4520 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:06.564365 4183 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:06.564860 4544 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:06.750913 3908 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250622 13:53:06.751602 4523 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:06.755766 3828 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250622 13:53:06.756844 4523 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:07.010305 4544 raft_consensus.cc:1062] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: attempting to promote NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 to VOTER
I20250622 13:53:07.011121 4544 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:07.013501 4259 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250622 13:53:07.013688 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250622 13:53:07.014276 4520 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:07.014873 4520 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:07.016414 3756 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250622 13:53:07.016980 4545 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:07.030503 4544 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.031455 3753 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.036267 3651 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.035986 4332 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.035653 4259 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.056095 3908 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:07.058507 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250622 13:53:07.058552 4259 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20250622 13:53:07.059067 4544 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:07.059909 4544 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:07.061215 4538 raft_consensus.cc:1062] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: attempting to promote NON_VOTER d23b47356efa46c792ab125b568ea95a to VOTER
I20250622 13:53:07.062012 4538 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:07.063802 4545 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.065012 4259 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.065013 4183 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 1 LEARNER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:07.066315 4523 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:07.068286 3828 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:07.068346 3908 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250622 13:53:07.068769 4523 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:07.070839 3655 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 reported cstate change: config changed from index 32 to 33, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.072194 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 91736fef6df543c4953497186dfe1c5e with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:07.074801 4558 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20250622 13:53:07.075583 4332 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:07.087224 3745 tablet_service.cc:1515] Processing DeleteTablet for tablet 91736fef6df543c4953497186dfe1c5e with delete_type TABLET_DATA_TOMBSTONED (TS d13e1e991fdc47899aac5b7c33fbbbd9 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:53772
I20250622 13:53:07.096439 4523 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:07.098006 3908 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:07.100018 3828 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:07.100597 4183 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:07.110954 3655 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index 28 to 29, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:07.114439 4562 tablet_replica.cc:331] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:07.114696 4562 raft_consensus.cc:2241] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:07.115748 4562 raft_consensus.cc:2270] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:07.116650 4562 ts_tablet_manager.cc:1905] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:07.117934 3753 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:07.120369 4183 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:07.121613 4523 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:07.123303 3908 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250622 13:53:07.124163 4523 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:07.125916 4523 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:07.127768 4562 ts_tablet_manager.cc:1918] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.32
I20250622 13:53:07.127869 4562 log.cc:1199] T 91736fef6df543c4953497186dfe1c5e P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root/wals/91736fef6df543c4953497186dfe1c5e
I20250622 13:53:07.128644 3644 catalog_manager.cc:4928] TS d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517): tablet 91736fef6df543c4953497186dfe1c5e (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:07.129470 3644 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:07.131402 3656 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:07.131646 3908 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:07.132892 4183 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:07.140527 3819 tablet_service.cc:1515] Processing DeleteTablet for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 with delete_type TABLET_DATA_TOMBSTONED (TS 946d416d6f4a4b0ea6a77b93cd6f58f6 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:45604
I20250622 13:53:07.141184 4563 tablet_replica.cc:331] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:07.141646 4563 raft_consensus.cc:2241] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:07.141882 4563 raft_consensus.cc:2270] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:07.143672 4563 ts_tablet_manager.cc:1905] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:07.151353 4563 ts_tablet_manager.cc:1918] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250622 13:53:07.151495 4563 log.cc:1199] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root/wals/5246e7fae8d44ba6a5e3ac826f9988f4
I20250622 13:53:07.152351 3643 catalog_manager.cc:4928] TS 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): tablet 5246e7fae8d44ba6a5e3ac826f9988f4 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:07.740293 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:07.742995 3756 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:07.743201 3756 raft_consensus.cc:604] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Received request to transfer leadership to TS 3377fd0584c546c1a938e3a1204b2be3
I20250622 13:53:07.744181 3756 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:07.744325 3756 raft_consensus.cc:604] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Received request to transfer leadership to TS 3377fd0584c546c1a938e3a1204b2be3
I20250622 13:53:07.745285 3908 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:53:07.745455 3908 raft_consensus.cc:604] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Received request to transfer leadership to TS 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:53:07.745877 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250622 13:53:07.746086 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:07.886559 4558 raft_consensus.cc:991] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:07.886729 4558 raft_consensus.cc:1079] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Signalling peer 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:07.887266 3908 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:36350
I20250622 13:53:07.887413 3908 raft_consensus.cc:491] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:07.887499 3908 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:07.889256 3908 raft_consensus.cc:513] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:07.889837 3908 leader_election.cc:290] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 2 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:53:07.890164 3756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:07.890316 3756 raft_consensus.cc:3053] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:07.890254 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:07.890394 3756 raft_consensus.cc:738] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:07.890450 3828 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:07.890628 3756 consensus_queue.cc:260] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:07.890875 3756 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:07.892338 3828 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 2.
I20250622 13:53:07.892688 3873 leader_election.cc:304] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, 946d416d6f4a4b0ea6a77b93cd6f58f6; no voters:
I20250622 13:53:07.892688 3756 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 2.
I20250622 13:53:07.892957 4520 raft_consensus.cc:2802] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:07.893296 4520 raft_consensus.cc:695] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:07.893550 4520 consensus_queue.cc:237] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:07.895891 3652 catalog_manager.cc:5582] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 1 to 2, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250622 13:53:08.055498 4565 raft_consensus.cc:991] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:08.055682 4565 raft_consensus.cc:1079] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Signalling peer 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:08.056357 3828 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:45620
I20250622 13:53:08.056505 3828 raft_consensus.cc:491] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:08.056571 3828 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:08.058917 3828 raft_consensus.cc:513] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:08.059402 3828 leader_election.cc:290] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 3 election: Requested vote from peers 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649), d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
I20250622 13:53:08.059844 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:08.059844 4183 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "d23b47356efa46c792ab125b568ea95a"
I20250622 13:53:08.060086 3908 raft_consensus.cc:3053] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:53:08.060096 4183 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:08.060187 3908 raft_consensus.cc:738] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:08.060452 3908 consensus_queue.cc:260] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:08.060864 3908 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:08.063184 4183 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 3.
I20250622 13:53:08.063568 3908 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 3.
I20250622 13:53:08.063699 3797 leader_election.cc:304] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d23b47356efa46c792ab125b568ea95a; no voters:
I20250622 13:53:08.064026 4432 raft_consensus.cc:2802] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:53:08.064378 4432 raft_consensus.cc:695] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:08.064648 4432 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:08.067364 3652 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 2 to 3, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 3 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:08.136276 4259 consensus_queue.cc:237] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:08.138960 3828 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250622 13:53:08.139211 3908 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250622 13:53:08.139534 4575 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:08.139837 4574 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:08.144670 4575 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:08.145255 3828 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
W20250622 13:53:08.146204 4223 consensus_peers.cc:487] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 -> Peer 52c21167280a4d669123bb1ed7675d79 (127.0.103.70:35833): Couldn't send request to peer 52c21167280a4d669123bb1ed7675d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 677dddfcdf864e7f9b997d196b58bc78. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:08.146217 3908 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:08.150171 3651 catalog_manager.cc:5582] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 reported cstate change: config changed from index 30 to 32, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New cstate: current_term: 2 leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:08.159880 4183 consensus_queue.cc:237] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:08.162777 3828 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250622 13:53:08.163416 4584 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:08.163874 3756 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250622 13:53:08.164327 4584 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:08.166390 4583 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:08.166949 3828 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
W20250622 13:53:08.171172 4150 consensus_peers.cc:487] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a -> Peer feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925): Couldn't send request to peer feaebd1ce3104a2fa3cc207fd3c23468. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f10daa2b314a4660bd36fa7902789a8e. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:08.172659 3656 catalog_manager.cc:5582] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index 30 to 32, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New cstate: current_term: 2 leader_uuid: "d23b47356efa46c792ab125b568ea95a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
W20250622 13:53:08.175895 3705 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20250622 13:53:08.179677 4564 raft_consensus.cc:991] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:08.179806 4564 raft_consensus.cc:1079] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Signalling peer 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:08.181515 3908 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:36350
I20250622 13:53:08.181713 3908 raft_consensus.cc:491] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:08.181823 3908 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:08.183846 3756 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:08.184257 3908 raft_consensus.cc:513] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:08.184870 3908 leader_election.cc:290] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 2 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
W20250622 13:53:08.185292 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c9260a075f104769bc81ca7eb0c76cb2, TS 3377fd0584c546c1a938e3a1204b2be3 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
W20250622 13:53:08.185392 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c9260a075f104769bc81ca7eb0c76cb2, TS 3377fd0584c546c1a938e3a1204b2be3 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:08.185439 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:08.185565 3753 raft_consensus.cc:3053] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:08.185628 3753 raft_consensus.cc:738] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:08.185863 3753 consensus_queue.cc:260] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:08.186081 3753 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:08.188184 3753 raft_consensus.cc:2466] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 2.
I20250622 13:53:08.188539 3874 leader_election.cc:304] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:08.185302 4183 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "d23b47356efa46c792ab125b568ea95a"
I20250622 13:53:08.188887 4565 raft_consensus.cc:2802] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:08.188896 4183 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:08.189014 4565 raft_consensus.cc:695] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:08.189251 4565 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:08.191864 4183 raft_consensus.cc:2466] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 2.
I20250622 13:53:08.192840 3652 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 1 to 2, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250622 13:53:08.196934 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1aa8fa3d766949b99d0538f8b76498d5, TS 946d416d6f4a4b0ea6a77b93cd6f58f6 -> TS d23b47356efa46c792ab125b568ea95a move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
W20250622 13:53:08.197010 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1aa8fa3d766949b99d0538f8b76498d5, TS 946d416d6f4a4b0ea6a77b93cd6f58f6 -> TS d23b47356efa46c792ab125b568ea95a move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
W20250622 13:53:08.200412 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59, TS d13e1e991fdc47899aac5b7c33fbbbd9 -> TS feaebd1ce3104a2fa3cc207fd3c23468 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:08.200503 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59, TS d13e1e991fdc47899aac5b7c33fbbbd9 -> TS feaebd1ce3104a2fa3cc207fd3c23468 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:08.204228 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c3c17332dc1c4acaa727cf9b2fb8ed8e, TS 946d416d6f4a4b0ea6a77b93cd6f58f6 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:08.204309 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c3c17332dc1c4acaa727cf9b2fb8ed8e, TS 946d416d6f4a4b0ea6a77b93cd6f58f6 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:08.373692 3828 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250622 13:53:08.374203 4565 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250622 13:53:08.377938 3753 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250622 13:53:08.378863 4520 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250622 13:53:08.477846 3908 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:08.478303 4432 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:08.482697 4184 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:08.483196 4432 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:08.557848 4598 ts_tablet_manager.cc:927] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: Initiating tablet copy from peer d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
I20250622 13:53:08.558511 4598 tablet_copy_client.cc:323] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:33811
I20250622 13:53:08.564034 4194 tablet_copy_service.cc:140] P d23b47356efa46c792ab125b568ea95a: Received BeginTabletCopySession request for tablet f10daa2b314a4660bd36fa7902789a8e from peer feaebd1ce3104a2fa3cc207fd3c23468 ({username='slave'} at 127.0.0.1:51752)
I20250622 13:53:08.564193 4194 tablet_copy_service.cc:161] P d23b47356efa46c792ab125b568ea95a: Beginning new tablet copy session on tablet f10daa2b314a4660bd36fa7902789a8e from peer feaebd1ce3104a2fa3cc207fd3c23468 at {username='slave'} at 127.0.0.1:51752: session id = feaebd1ce3104a2fa3cc207fd3c23468-f10daa2b314a4660bd36fa7902789a8e
I20250622 13:53:08.566433 4194 tablet_copy_source_session.cc:215] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:08.567485 4598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10daa2b314a4660bd36fa7902789a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:08.572590 4598 tablet_copy_client.cc:806] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:08.572832 4598 tablet_copy_client.cc:670] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:08.574173 4599 tablet_copy_client.cc:1025] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:08.599100 4600 ts_tablet_manager.cc:927] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: Initiating tablet copy from peer feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925)
I20250622 13:53:08.600416 4600 tablet_copy_client.cc:323] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:42925
I20250622 13:53:08.606491 4269 tablet_copy_service.cc:140] P feaebd1ce3104a2fa3cc207fd3c23468: Received BeginTabletCopySession request for tablet 677dddfcdf864e7f9b997d196b58bc78 from peer 52c21167280a4d669123bb1ed7675d79 ({username='slave'} at 127.0.0.1:47684)
I20250622 13:53:08.606649 4269 tablet_copy_service.cc:161] P feaebd1ce3104a2fa3cc207fd3c23468: Beginning new tablet copy session on tablet 677dddfcdf864e7f9b997d196b58bc78 from peer 52c21167280a4d669123bb1ed7675d79 at {username='slave'} at 127.0.0.1:47684: session id = 52c21167280a4d669123bb1ed7675d79-677dddfcdf864e7f9b997d196b58bc78
I20250622 13:53:08.608677 3753 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:08.608991 4269 tablet_copy_source_session.cc:215] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:08.609166 4570 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:08.610152 4600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 677dddfcdf864e7f9b997d196b58bc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:08.618551 4600 tablet_copy_client.cc:806] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:08.618949 4600 tablet_copy_client.cc:670] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:08.620666 4601 tablet_copy_client.cc:1025] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:08.626763 4183 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:08.628885 4570 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:09.746409 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:09.749624 3908 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "91736fef6df543c4953497186dfe1c5e"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:53:09.749852 3908 raft_consensus.cc:604] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Received request to transfer leadership to TS feaebd1ce3104a2fa3cc207fd3c23468
I20250622 13:53:09.750867 3908 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:53:09.751055 3908 raft_consensus.cc:604] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:53:09.752051 3828 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:09.752221 3828 raft_consensus.cc:604] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Received request to transfer leadership to TS 3377fd0584c546c1a938e3a1204b2be3
I20250622 13:53:09.752660 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250622 13:53:09.752911 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:09.780475 4612 raft_consensus.cc:991] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:09.780673 4612 raft_consensus.cc:1079] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Signalling peer 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:09.781227 3908 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:36344
I20250622 13:53:09.781430 3908 raft_consensus.cc:491] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:09.781533 3908 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:09.783497 3908 raft_consensus.cc:513] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:09.784045 3908 leader_election.cc:290] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 4 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
I20250622 13:53:09.784458 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:09.784533 4184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9260a075f104769bc81ca7eb0c76cb2" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "d23b47356efa46c792ab125b568ea95a"
I20250622 13:53:09.784629 3828 raft_consensus.cc:3053] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Stepping down as leader of term 3
I20250622 13:53:09.784633 4184 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:09.784710 3828 raft_consensus.cc:738] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:09.784984 3828 consensus_queue.cc:260] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:09.785280 3828 raft_consensus.cc:3058] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:09.787345 4184 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 4.
I20250622 13:53:09.787703 3871 leader_election.cc:304] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d23b47356efa46c792ab125b568ea95a; no voters:
I20250622 13:53:09.787757 3828 raft_consensus.cc:2466] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 4.
I20250622 13:53:09.787961 4570 raft_consensus.cc:2802] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 13:53:09.788529 4570 raft_consensus.cc:695] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:09.788800 4570 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:09.791203 3656 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 3 to 4, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 4 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:09.876334 4613 raft_consensus.cc:991] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower feaebd1ce3104a2fa3cc207fd3c23468 to start an election
I20250622 13:53:09.876510 4613 raft_consensus.cc:1079] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Signalling peer feaebd1ce3104a2fa3cc207fd3c23468 to start an election
I20250622 13:53:09.877027 4259 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "91736fef6df543c4953497186dfe1c5e"
dest_uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
from {username='slave'} at 127.0.0.1:33372
I20250622 13:53:09.877198 4259 raft_consensus.cc:491] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:09.877310 4259 raft_consensus.cc:3058] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:09.879539 4259 raft_consensus.cc:513] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:09.880429 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91736fef6df543c4953497186dfe1c5e" candidate_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:09.880594 3908 raft_consensus.cc:3053] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:09.880587 4332 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91736fef6df543c4953497186dfe1c5e" candidate_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: true dest_uuid: "52c21167280a4d669123bb1ed7675d79"
I20250622 13:53:09.880704 3908 raft_consensus.cc:738] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:09.880759 4332 raft_consensus.cc:3058] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:09.880937 3908 consensus_queue.cc:260] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 33, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:09.881220 3908 raft_consensus.cc:3058] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:09.883558 4332 raft_consensus.cc:2466] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feaebd1ce3104a2fa3cc207fd3c23468 in term 2.
I20250622 13:53:09.883751 3908 raft_consensus.cc:2466] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feaebd1ce3104a2fa3cc207fd3c23468 in term 2.
I20250622 13:53:09.884008 4223 leader_election.cc:304] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c21167280a4d669123bb1ed7675d79, feaebd1ce3104a2fa3cc207fd3c23468; no voters:
I20250622 13:53:09.880606 4259 leader_election.cc:290] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [CANDIDATE]: Term 2 election: Requested vote from peers 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649), 52c21167280a4d669123bb1ed7675d79 (127.0.103.70:35833)
I20250622 13:53:09.884284 4611 raft_consensus.cc:2802] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:09.884403 4611 raft_consensus.cc:695] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Becoming Leader. State: Replica: feaebd1ce3104a2fa3cc207fd3c23468, State: Running, Role: LEADER
I20250622 13:53:09.884616 4611 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:09.887108 3656 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 reported cstate change: term changed from 1 to 2, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69). New cstate: current_term: 2 leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:09.900310 4570 raft_consensus.cc:991] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:09.900476 4570 raft_consensus.cc:1079] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Signalling peer d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:09.901086 3753 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:53794
I20250622 13:53:09.901299 3753 raft_consensus.cc:491] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:09.901443 3753 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:09.904065 3753 raft_consensus.cc:513] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:09.904574 3753 leader_election.cc:290] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 3 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:09.904982 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:09.904978 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:09.905169 3828 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:09.905227 3908 raft_consensus.cc:3053] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:53:09.905290 3908 raft_consensus.cc:738] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:09.905531 3908 consensus_queue.cc:260] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:09.905762 3908 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:09.907758 3828 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 3.
I20250622 13:53:09.908089 3908 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 3.
I20250622 13:53:09.908133 3724 leader_election.cc:304] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:09.908471 4564 raft_consensus.cc:2802] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:53:09.909476 4564 raft_consensus.cc:695] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:09.909749 4564 consensus_queue.cc:237] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:09.912225 3656 catalog_manager.cc:5582] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 2 to 3, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 3 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:10.200080 4184 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250622 13:53:10.200563 4570 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:10.232852 3828 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250622 13:53:10.233759 4613 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:10.332208 3908 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250622 13:53:10.332763 4611 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:10.337991 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250622 13:53:10.338665 4611 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:10.385834 3828 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250622 13:53:10.386356 4564 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:10.390834 3908 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250622 13:53:10.391548 4620 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250622 13:53:10.574551 4598 tablet_copy_client.cc:538] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:10.577920 4598 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap starting.
I20250622 13:53:10.621150 4600 tablet_copy_client.cc:538] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:10.624450 4600 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: Bootstrap starting.
I20250622 13:53:10.643497 4598 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:10.643887 4598 tablet_bootstrap.cc:492] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap complete.
I20250622 13:53:10.644146 4598 ts_tablet_manager.cc:1397] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: Time spent bootstrapping tablet: real 0.066s user 0.054s sys 0.009s
I20250622 13:53:10.644861 4598 raft_consensus.cc:357] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:10.645179 4598 raft_consensus.cc:738] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: feaebd1ce3104a2fa3cc207fd3c23468, State: Initialized, Role: LEARNER
I20250622 13:53:10.645454 4598 consensus_queue.cc:260] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:10.646433 4598 ts_tablet_manager.cc:1428] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250622 13:53:10.646987 4194 tablet_copy_service.cc:342] P d23b47356efa46c792ab125b568ea95a: Request end of tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-f10daa2b314a4660bd36fa7902789a8e received from {username='slave'} at 127.0.0.1:51752
I20250622 13:53:10.647110 4194 tablet_copy_service.cc:434] P d23b47356efa46c792ab125b568ea95a: ending tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-f10daa2b314a4660bd36fa7902789a8e on tablet f10daa2b314a4660bd36fa7902789a8e with peer feaebd1ce3104a2fa3cc207fd3c23468
I20250622 13:53:10.670980 4600 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=99 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:10.671379 4600 tablet_bootstrap.cc:492] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: Bootstrap complete.
I20250622 13:53:10.671614 4600 ts_tablet_manager.cc:1397] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: Time spent bootstrapping tablet: real 0.047s user 0.036s sys 0.011s
I20250622 13:53:10.672328 4600 raft_consensus.cc:357] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:10.672489 4600 raft_consensus.cc:738] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 52c21167280a4d669123bb1ed7675d79, State: Initialized, Role: LEARNER
I20250622 13:53:10.672667 4600 consensus_queue.cc:260] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:10.673189 4600 ts_tablet_manager.cc:1428] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:53:10.673820 4269 tablet_copy_service.cc:342] P feaebd1ce3104a2fa3cc207fd3c23468: Request end of tablet copy session 52c21167280a4d669123bb1ed7675d79-677dddfcdf864e7f9b997d196b58bc78 received from {username='slave'} at 127.0.0.1:47684
I20250622 13:53:10.673929 4269 tablet_copy_service.cc:434] P feaebd1ce3104a2fa3cc207fd3c23468: ending tablet copy session 52c21167280a4d669123bb1ed7675d79-677dddfcdf864e7f9b997d196b58bc78 on tablet 677dddfcdf864e7f9b997d196b58bc78 with peer 52c21167280a4d669123bb1ed7675d79
I20250622 13:53:10.750061 4332 raft_consensus.cc:1215] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20250622 13:53:11.083899 4259 raft_consensus.cc:1215] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20250622 13:53:11.149832 4610 raft_consensus.cc:1062] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: attempting to promote NON_VOTER 52c21167280a4d669123bb1ed7675d79 to VOTER
I20250622 13:53:11.150827 4610 consensus_queue.cc:237] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:11.153012 4332 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 LEARNER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250622 13:53:11.153034 3828 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250622 13:53:11.153034 3908 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250622 13:53:11.153659 4637 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:11.153988 4611 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:11.154521 4637 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:11.159670 4637 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:11.160301 3908 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:11.163425 4332 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:11.164412 3652 catalog_manager.cc:5582] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 reported cstate change: config changed from index 32 to 33, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:11.164866 3828 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:11.178196 4258 consensus_queue.cc:237] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:11.179764 3828 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250622 13:53:11.179898 4332 raft_consensus.cc:1273] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250622 13:53:11.180328 4641 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:11.180891 4637 consensus_queue.cc:1035] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:11.182379 4641 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:11.183478 4332 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:11.185599 3704 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 677dddfcdf864e7f9b997d196b58bc78 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:11.185995 3828 raft_consensus.cc:2953] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:11.186678 3655 catalog_manager.cc:5582] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 reported cstate change: config changed from index 33 to 34, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:11.190454 3890 tablet_service.cc:1515] Processing DeleteTablet for tablet 677dddfcdf864e7f9b997d196b58bc78 with delete_type TABLET_DATA_TOMBSTONED (TS 3377fd0584c546c1a938e3a1204b2be3 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:36332
I20250622 13:53:11.191223 4649 tablet_replica.cc:331] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:11.191431 4649 raft_consensus.cc:2241] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:11.191695 4649 raft_consensus.cc:2270] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:11.192517 4649 ts_tablet_manager.cc:1905] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:11.198005 4649 ts_tablet_manager.cc:1918] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20250622 13:53:11.198118 4649 log.cc:1199] T 677dddfcdf864e7f9b997d196b58bc78 P 3377fd0584c546c1a938e3a1204b2be3: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root/wals/677dddfcdf864e7f9b997d196b58bc78
I20250622 13:53:11.198776 3643 catalog_manager.cc:4928] TS 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): tablet 677dddfcdf864e7f9b997d196b58bc78 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:11.561416 4651 raft_consensus.cc:1062] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: attempting to promote NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 to VOTER
I20250622 13:53:11.562170 4651 consensus_queue.cc:237] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:11.564296 3828 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250622 13:53:11.564757 3753 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250622 13:53:11.564813 4258 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEARNER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250622 13:53:11.565774 4608 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:11.566105 4654 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:11.566620 4650 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:11.570853 4608 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:11.576292 3828 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:11.577353 3655 catalog_manager.cc:5582] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 32 to 33, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d23b47356efa46c792ab125b568ea95a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:11.580247 4258 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:11.580878 3753 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:11.587193 4183 consensus_queue.cc:237] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:11.588733 4258 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250622 13:53:11.588968 3828 raft_consensus.cc:1273] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250622 13:53:11.589466 4651 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:11.589716 4650 consensus_queue.cc:1035] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:11.591787 4651 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:11.592720 3828 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:11.596293 3699 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f10daa2b314a4660bd36fa7902789a8e with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:11.592690 4258 raft_consensus.cc:2953] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:11.597779 3656 catalog_manager.cc:5582] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 33 to 34, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "d23b47356efa46c792ab125b568ea95a" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:11.601259 3745 tablet_service.cc:1515] Processing DeleteTablet for tablet f10daa2b314a4660bd36fa7902789a8e with delete_type TABLET_DATA_TOMBSTONED (TS d13e1e991fdc47899aac5b7c33fbbbd9 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:53772
I20250622 13:53:11.602152 4658 tablet_replica.cc:331] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:11.602339 4658 raft_consensus.cc:2241] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:11.602559 4658 raft_consensus.cc:2270] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:11.603297 4658 ts_tablet_manager.cc:1905] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:11.607470 4658 ts_tablet_manager.cc:1918] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20250622 13:53:11.607606 4658 log.cc:1199] T f10daa2b314a4660bd36fa7902789a8e P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root/wals/f10daa2b314a4660bd36fa7902789a8e
I20250622 13:53:11.608256 3644 catalog_manager.cc:4928] TS d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517): tablet f10daa2b314a4660bd36fa7902789a8e (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:11.753336 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:11.756244 3908 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "d23b47356efa46c792ab125b568ea95a"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:53:11.756467 3908 raft_consensus.cc:604] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Received request to transfer leadership to TS d23b47356efa46c792ab125b568ea95a
I20250622 13:53:11.757515 3753 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:11.757679 3753 raft_consensus.cc:604] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Received request to transfer leadership to TS 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:53:11.758101 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250622 13:53:11.758327 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:12.033983 4639 raft_consensus.cc:991] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:12.034133 4639 raft_consensus.cc:1079] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Signalling peer 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:12.034734 3828 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:45606
I20250622 13:53:12.034902 3828 raft_consensus.cc:491] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:12.034986 3828 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:12.037580 3828 raft_consensus.cc:513] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:12.038298 3828 leader_election.cc:290] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 4 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:12.038736 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:12.038815 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:12.038902 3753 raft_consensus.cc:3053] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Stepping down as leader of term 3
I20250622 13:53:12.038987 3753 raft_consensus.cc:738] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:12.039005 3908 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:12.039238 3753 consensus_queue.cc:260] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:12.039531 3753 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:12.041229 3753 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 4.
I20250622 13:53:12.041638 3800 leader_election.cc:304] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:12.041893 4612 raft_consensus.cc:2802] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 13:53:12.042253 4612 raft_consensus.cc:695] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:12.042542 4612 consensus_queue.cc:237] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:12.044917 3655 catalog_manager.cc:5582] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 3 to 4, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 4 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:12.056354 3908 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 4.
I20250622 13:53:12.153277 4570 raft_consensus.cc:991] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower d23b47356efa46c792ab125b568ea95a to start an election
I20250622 13:53:12.153493 4662 raft_consensus.cc:1079] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Signalling peer d23b47356efa46c792ab125b568ea95a to start an election
I20250622 13:53:12.154105 4184 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4"
dest_uuid: "d23b47356efa46c792ab125b568ea95a"
from {username='slave'} at 127.0.0.1:51734
I20250622 13:53:12.154274 4184 raft_consensus.cc:491] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:12.154368 4184 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:12.156221 4184 raft_consensus.cc:513] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:12.156844 4184 leader_election.cc:290] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [CANDIDATE]: Term 3 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:12.157277 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4" candidate_uuid: "d23b47356efa46c792ab125b568ea95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:12.157481 3753 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:12.160056 3753 raft_consensus.cc:2466] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23b47356efa46c792ab125b568ea95a in term 3.
I20250622 13:53:12.160457 4150 leader_election.cc:304] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13e1e991fdc47899aac5b7c33fbbbd9, d23b47356efa46c792ab125b568ea95a; no voters:
I20250622 13:53:12.160741 4650 raft_consensus.cc:2802] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:53:12.161126 4650 raft_consensus.cc:695] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 3 LEADER]: Becoming Leader. State: Replica: d23b47356efa46c792ab125b568ea95a, State: Running, Role: LEADER
I20250622 13:53:12.161463 4650 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:12.163987 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5246e7fae8d44ba6a5e3ac826f9988f4" candidate_uuid: "d23b47356efa46c792ab125b568ea95a" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:12.164245 3908 raft_consensus.cc:3053] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:53:12.164316 3908 raft_consensus.cc:738] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:12.164160 3655 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a reported cstate change: term changed from 2 to 3, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to d23b47356efa46c792ab125b568ea95a (127.0.103.68). New cstate: current_term: 3 leader_uuid: "d23b47356efa46c792ab125b568ea95a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:12.164561 3908 consensus_queue.cc:260] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:12.164883 3908 raft_consensus.cc:3058] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:12.167091 3908 raft_consensus.cc:2466] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23b47356efa46c792ab125b568ea95a in term 3.
I20250622 13:53:12.512445 3908 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250622 13:53:12.512975 4612 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:12.516825 3753 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250622 13:53:12.517519 4660 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:12.603019 3908 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:12.605648 4184 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20250622 13:53:12.605695 3828 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
W20250622 13:53:12.605854 3872 consensus_peers.cc:487] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 -> Peer 52c21167280a4d669123bb1ed7675d79 (127.0.103.70:35833): Couldn't send request to peer 52c21167280a4d669123bb1ed7675d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9260a075f104769bc81ca7eb0c76cb2. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:12.606168 4674 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:12.606438 4570 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:12.609061 4674 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 30 to 34, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:12.609697 4184 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 30 to 34, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:12.610183 3828 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 30 to 34, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:12.614614 3651 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 30 to 34, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New cstate: current_term: 4 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:12.616127 3753 consensus_queue.cc:237] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
W20250622 13:53:12.619239 3722 consensus_peers.cc:487] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 -> Peer d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811): Couldn't send request to peer d23b47356efa46c792ab125b568ea95a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b045aeb5889a4b88b8ee72370f2a7e39. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:12.619328 3828 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250622 13:53:12.619844 4678 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:12.620319 3908 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250622 13:53:12.621018 4678 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:12.622457 4639 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:12.625489 3828 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:12.626096 3908 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Committing config change with OpId 4.30: config changed from index -1 to 30, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:12.627108 3656 catalog_manager.cc:5582] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index -1 to 30, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New cstate: current_term: 4 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:12.629156 4258 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:12.641114 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
W20250622 13:53:12.641227 4222 consensus_peers.cc:487] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 -> Peer d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811): Couldn't send request to peer d23b47356efa46c792ab125b568ea95a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91736fef6df543c4953497186dfe1c5e. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:12.641870 4610 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 13:53:12.641114 3908 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20250622 13:53:12.642997 4610 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 13:53:12.644439 4685 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 33 to 35, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:12.645099 4332 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 33 to 35, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:12.646024 3908 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 33 to 35, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
W20250622 13:53:12.649060 3705 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20250622 13:53:12.649123 3656 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 reported cstate change: config changed from index 33 to 35, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250622 13:53:12.655109 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5246e7fae8d44ba6a5e3ac826f9988f4, TS d13e1e991fdc47899aac5b7c33fbbbd9 -> TS feaebd1ce3104a2fa3cc207fd3c23468 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
W20250622 13:53:12.655191 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5246e7fae8d44ba6a5e3ac826f9988f4, TS d13e1e991fdc47899aac5b7c33fbbbd9 -> TS feaebd1ce3104a2fa3cc207fd3c23468 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
W20250622 13:53:12.659243 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bac2340a96e94c219f59c12945eb8bf6, TS 946d416d6f4a4b0ea6a77b93cd6f58f6 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:12.659343 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet bac2340a96e94c219f59c12945eb8bf6, TS 946d416d6f4a4b0ea6a77b93cd6f58f6 -> TS 52c21167280a4d669123bb1ed7675d79 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:12.663759 3705 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c3c17332dc1c4acaa727cf9b2fb8ed8e, TS 3377fd0584c546c1a938e3a1204b2be3 -> TS feaebd1ce3104a2fa3cc207fd3c23468 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
W20250622 13:53:12.663872 3705 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c3c17332dc1c4acaa727cf9b2fb8ed8e, TS 3377fd0584c546c1a938e3a1204b2be3 -> TS feaebd1ce3104a2fa3cc207fd3c23468 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:12.693208 3753 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:12.693902 4650 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:12.705439 3903 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:12.706081 4689 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:13.150476 4692 ts_tablet_manager.cc:927] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: Initiating tablet copy from peer d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517)
I20250622 13:53:13.151324 4692 tablet_copy_client.cc:323] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:40517
I20250622 13:53:13.152014 3769 tablet_copy_service.cc:140] P d13e1e991fdc47899aac5b7c33fbbbd9: Received BeginTabletCopySession request for tablet b045aeb5889a4b88b8ee72370f2a7e39 from peer d23b47356efa46c792ab125b568ea95a ({username='slave'} at 127.0.0.1:53832)
I20250622 13:53:13.152190 3769 tablet_copy_service.cc:161] P d13e1e991fdc47899aac5b7c33fbbbd9: Beginning new tablet copy session on tablet b045aeb5889a4b88b8ee72370f2a7e39 from peer d23b47356efa46c792ab125b568ea95a at {username='slave'} at 127.0.0.1:53832: session id = d23b47356efa46c792ab125b568ea95a-b045aeb5889a4b88b8ee72370f2a7e39
I20250622 13:53:13.154564 3769 tablet_copy_source_session.cc:215] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:13.155676 4692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b045aeb5889a4b88b8ee72370f2a7e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:13.160094 4692 tablet_copy_client.cc:806] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:13.160327 4692 tablet_copy_client.cc:670] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:13.161931 4693 tablet_copy_client.cc:1025] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:13.191385 4695 ts_tablet_manager.cc:927] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: Initiating tablet copy from peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:13.192049 4695 tablet_copy_client.cc:323] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33649
I20250622 13:53:13.192713 3918 tablet_copy_service.cc:140] P 3377fd0584c546c1a938e3a1204b2be3: Received BeginTabletCopySession request for tablet c9260a075f104769bc81ca7eb0c76cb2 from peer 52c21167280a4d669123bb1ed7675d79 ({username='slave'} at 127.0.0.1:36376)
I20250622 13:53:13.192893 3918 tablet_copy_service.cc:161] P 3377fd0584c546c1a938e3a1204b2be3: Beginning new tablet copy session on tablet c9260a075f104769bc81ca7eb0c76cb2 from peer 52c21167280a4d669123bb1ed7675d79 at {username='slave'} at 127.0.0.1:36376: session id = 52c21167280a4d669123bb1ed7675d79-c9260a075f104769bc81ca7eb0c76cb2
I20250622 13:53:13.194792 4694 ts_tablet_manager.cc:927] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: Initiating tablet copy from peer feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925)
I20250622 13:53:13.195410 4694 tablet_copy_client.cc:323] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:42925
I20250622 13:53:13.195521 3918 tablet_copy_source_session.cc:215] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:13.196122 4269 tablet_copy_service.cc:140] P feaebd1ce3104a2fa3cc207fd3c23468: Received BeginTabletCopySession request for tablet 91736fef6df543c4953497186dfe1c5e from peer d23b47356efa46c792ab125b568ea95a ({username='slave'} at 127.0.0.1:47674)
I20250622 13:53:13.196280 4269 tablet_copy_service.cc:161] P feaebd1ce3104a2fa3cc207fd3c23468: Beginning new tablet copy session on tablet 91736fef6df543c4953497186dfe1c5e from peer d23b47356efa46c792ab125b568ea95a at {username='slave'} at 127.0.0.1:47674: session id = d23b47356efa46c792ab125b568ea95a-91736fef6df543c4953497186dfe1c5e
I20250622 13:53:13.198549 4269 tablet_copy_source_session.cc:215] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:13.199589 4695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9260a075f104769bc81ca7eb0c76cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:13.201840 4694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91736fef6df543c4953497186dfe1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:13.204859 4695 tablet_copy_client.cc:806] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:13.205148 4695 tablet_copy_client.cc:670] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:13.206871 4696 tablet_copy_client.cc:1025] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:13.207659 4694 tablet_copy_client.cc:806] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:13.207863 4694 tablet_copy_client.cc:670] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:13.209244 4697 tablet_copy_client.cc:1025] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:13.758661 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:13.762360 3828 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:13.762603 3828 raft_consensus.cc:604] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 LEADER]: Received request to transfer leadership to TS 3377fd0584c546c1a938e3a1204b2be3
I20250622 13:53:13.763106 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:13.763321 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:14.111129 4706 raft_consensus.cc:991] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:14.111286 4706 raft_consensus.cc:1079] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 LEADER]: Signalling peer 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:14.112059 3903 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:36344
I20250622 13:53:14.112258 3903 raft_consensus.cc:491] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:14.112360 3903 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:14.114591 3903 raft_consensus.cc:513] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:14.115254 3903 leader_election.cc:290] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 5 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:53:14.115684 3756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:14.115720 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3c17332dc1c4acaa727cf9b2fb8ed8e" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:14.115867 3756 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:14.115885 3828 raft_consensus.cc:3053] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 LEADER]: Stepping down as leader of term 4
I20250622 13:53:14.115976 3828 raft_consensus.cc:738] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:14.116214 3828 consensus_queue.cc:260] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:14.116478 3828 raft_consensus.cc:3058] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:14.118418 3756 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 5.
I20250622 13:53:14.118558 3828 raft_consensus.cc:2466] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 5.
I20250622 13:53:14.118814 3874 leader_election.cc:304] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:14.119091 4704 raft_consensus.cc:2802] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 5 FOLLOWER]: Leader election won for term 5
I20250622 13:53:14.119220 4704 raft_consensus.cc:695] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 5 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:14.119439 4704 consensus_queue.cc:237] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } }
I20250622 13:53:14.121862 3655 catalog_manager.cc:5582] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 4 to 5, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 5 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250622 13:53:14.615206 3753 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20250622 13:53:14.615674 4705 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:14.620937 3828 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20250622 13:53:14.621466 4705 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:15.162313 4692 tablet_copy_client.cc:538] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:15.165242 4692 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: Bootstrap starting.
I20250622 13:53:15.200902 4692 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:15.201329 4692 tablet_bootstrap.cc:492] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: Bootstrap complete.
I20250622 13:53:15.201617 4692 ts_tablet_manager.cc:1397] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: Time spent bootstrapping tablet: real 0.036s user 0.030s sys 0.008s
I20250622 13:53:15.202250 4692 raft_consensus.cc:357] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:15.202437 4692 raft_consensus.cc:738] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: d23b47356efa46c792ab125b568ea95a, State: Initialized, Role: LEARNER
I20250622 13:53:15.202629 4692 consensus_queue.cc:260] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:15.203094 4692 ts_tablet_manager.cc:1428] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:53:15.203704 3769 tablet_copy_service.cc:342] P d13e1e991fdc47899aac5b7c33fbbbd9: Request end of tablet copy session d23b47356efa46c792ab125b568ea95a-b045aeb5889a4b88b8ee72370f2a7e39 received from {username='slave'} at 127.0.0.1:53832
I20250622 13:53:15.203796 3769 tablet_copy_service.cc:434] P d13e1e991fdc47899aac5b7c33fbbbd9: ending tablet copy session d23b47356efa46c792ab125b568ea95a-b045aeb5889a4b88b8ee72370f2a7e39 on tablet b045aeb5889a4b88b8ee72370f2a7e39 with peer d23b47356efa46c792ab125b568ea95a
I20250622 13:53:15.207412 4695 tablet_copy_client.cc:538] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:15.209559 4694 tablet_copy_client.cc:538] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:15.210528 4695 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: Bootstrap starting.
I20250622 13:53:15.212494 4694 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: Bootstrap starting.
I20250622 13:53:15.250329 4694 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:15.250681 4694 tablet_bootstrap.cc:492] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: Bootstrap complete.
I20250622 13:53:15.250917 4694 ts_tablet_manager.cc:1397] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: Time spent bootstrapping tablet: real 0.038s user 0.028s sys 0.009s
I20250622 13:53:15.251518 4694 raft_consensus.cc:357] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:15.251732 4694 raft_consensus.cc:738] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d23b47356efa46c792ab125b568ea95a, State: Initialized, Role: LEARNER
I20250622 13:53:15.251960 4694 consensus_queue.cc:260] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:15.252538 4694 ts_tablet_manager.cc:1428] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250622 13:53:15.253291 4269 tablet_copy_service.cc:342] P feaebd1ce3104a2fa3cc207fd3c23468: Request end of tablet copy session d23b47356efa46c792ab125b568ea95a-91736fef6df543c4953497186dfe1c5e received from {username='slave'} at 127.0.0.1:47674
I20250622 13:53:15.253420 4269 tablet_copy_service.cc:434] P feaebd1ce3104a2fa3cc207fd3c23468: ending tablet copy session d23b47356efa46c792ab125b568ea95a-91736fef6df543c4953497186dfe1c5e on tablet 91736fef6df543c4953497186dfe1c5e with peer d23b47356efa46c792ab125b568ea95a
I20250622 13:53:15.256510 4695 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:15.256945 4695 tablet_bootstrap.cc:492] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: Bootstrap complete.
I20250622 13:53:15.257208 4695 ts_tablet_manager.cc:1397] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: Time spent bootstrapping tablet: real 0.047s user 0.042s sys 0.000s
I20250622 13:53:15.257928 4695 raft_consensus.cc:357] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:15.258158 4695 raft_consensus.cc:738] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 52c21167280a4d669123bb1ed7675d79, State: Initialized, Role: LEARNER
I20250622 13:53:15.258354 4695 consensus_queue.cc:260] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:15.258862 4695 ts_tablet_manager.cc:1428] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:15.259485 3918 tablet_copy_service.cc:342] P 3377fd0584c546c1a938e3a1204b2be3: Request end of tablet copy session 52c21167280a4d669123bb1ed7675d79-c9260a075f104769bc81ca7eb0c76cb2 received from {username='slave'} at 127.0.0.1:36376
I20250622 13:53:15.259579 3918 tablet_copy_service.cc:434] P 3377fd0584c546c1a938e3a1204b2be3: ending tablet copy session 52c21167280a4d669123bb1ed7675d79-c9260a075f104769bc81ca7eb0c76cb2 on tablet c9260a075f104769bc81ca7eb0c76cb2 with peer 52c21167280a4d669123bb1ed7675d79
I20250622 13:53:15.261881 4184 raft_consensus.cc:1215] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.34->[2.35-2.35] Dedup: 2.35->[]
I20250622 13:53:15.500757 4332 raft_consensus.cc:1215] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.33->[4.34-4.34] Dedup: 4.34->[]
I20250622 13:53:15.534169 4184 raft_consensus.cc:1215] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 4 LEARNER]: Deduplicated request from leader. Original: 4.29->[4.30-4.30] Dedup: 4.30->[]
I20250622 13:53:15.755673 4723 raft_consensus.cc:1062] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: attempting to promote NON_VOTER d23b47356efa46c792ab125b568ea95a to VOTER
I20250622 13:53:15.756608 4723 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 2.35, Last appended by leader: 33, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:15.759052 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250622 13:53:15.759181 3903 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250622 13:53:15.759675 4712 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250622 13:53:15.760088 4723 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250622 13:53:15.760658 4184 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 LEARNER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20250622 13:53:15.761194 4725 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250622 13:53:15.763810 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:15.763470 4723 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Committing config change with OpId 2.36: config changed from index 35 to 36, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.765235 4184 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.765849 4332 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.769210 3903 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:53:15.769404 3903 raft_consensus.cc:604] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:53:15.769986 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:15.770202 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:15.770411 3651 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 reported cstate change: config changed from index 35 to 36, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.772091 3903 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Committing config change with OpId 2.36: config changed from index 35 to 36, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.780985 4259 consensus_queue.cc:237] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:15.782766 4184 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250622 13:53:15.782879 4332 raft_consensus.cc:1273] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Refusing update from remote peer feaebd1ce3104a2fa3cc207fd3c23468: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20250622 13:53:15.783344 4725 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250622 13:53:15.783787 4722 consensus_queue.cc:1035] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250622 13:53:15.785965 4723 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.786612 4184 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.786795 4332 raft_consensus.cc:2953] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index 36 to 37, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.789741 3704 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 91736fef6df543c4953497186dfe1c5e with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:15.790843 3652 catalog_manager.cc:5582] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 reported cstate change: config changed from index 36 to 37, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:15.795508 3890 tablet_service.cc:1515] Processing DeleteTablet for tablet 91736fef6df543c4953497186dfe1c5e with delete_type TABLET_DATA_TOMBSTONED (TS 3377fd0584c546c1a938e3a1204b2be3 not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:36332
I20250622 13:53:15.797880 4733 tablet_replica.cc:331] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:15.798072 4733 raft_consensus.cc:2241] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:15.798316 4733 raft_consensus.cc:2270] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:15.799072 4733 ts_tablet_manager.cc:1905] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:15.804322 4733 ts_tablet_manager.cc:1918] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.36
I20250622 13:53:15.804454 4733 log.cc:1199] T 91736fef6df543c4953497186dfe1c5e P 3377fd0584c546c1a938e3a1204b2be3: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root/wals/91736fef6df543c4953497186dfe1c5e
I20250622 13:53:15.805155 3643 catalog_manager.cc:4928] TS 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): tablet 91736fef6df543c4953497186dfe1c5e (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:15.808640 4734 raft_consensus.cc:991] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:15.808769 4734 raft_consensus.cc:1079] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Signalling peer d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:15.809298 3753 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:53794
I20250622 13:53:15.809500 3753 raft_consensus.cc:491] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:15.809597 3753 raft_consensus.cc:3058] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:15.811735 3753 raft_consensus.cc:513] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:15.812410 3753 leader_election.cc:290] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 2 election: Requested vote from peers 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649), feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925)
I20250622 13:53:15.812705 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:15.812867 3908 raft_consensus.cc:3053] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:15.812944 3908 raft_consensus.cc:738] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:15.813199 3908 consensus_queue.cc:260] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:15.813547 3908 raft_consensus.cc:3058] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:15.815981 3908 raft_consensus.cc:2466] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 2.
I20250622 13:53:15.816412 3724 leader_election.cc:304] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:15.816646 4678 raft_consensus.cc:2802] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:15.817024 4678 raft_consensus.cc:695] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:15.817296 4678 consensus_queue.cc:237] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:15.819224 4259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d317ca8305314d45b47fa1014f3b5162" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
I20250622 13:53:15.819382 4259 raft_consensus.cc:3058] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:15.819598 3655 catalog_manager.cc:5582] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 1 to 2, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:15.821746 4259 raft_consensus.cc:2466] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 2.
I20250622 13:53:15.967239 4678 raft_consensus.cc:1062] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: attempting to promote NON_VOTER d23b47356efa46c792ab125b568ea95a to VOTER
I20250622 13:53:15.967859 4678 consensus_queue.cc:237] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 28, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:15.970603 4184 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 4 LEARNER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250622 13:53:15.970660 3903 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250622 13:53:15.970706 3828 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250622 13:53:15.971092 4739 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:15.971323 4737 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:15.971717 4737 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:15.979691 4678 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.980636 3828 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.983489 3903 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.984804 4184 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.985219 3655 catalog_manager.cc:5582] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:15.990942 3753 consensus_queue.cc:237] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:15.992457 4184 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250622 13:53:15.992919 4678 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:15.994310 3828 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250622 13:53:15.994843 4678 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:15.995388 4678 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.995849 4184 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:15.999600 3644 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b045aeb5889a4b88b8ee72370f2a7e39 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:16.000119 3656 catalog_manager.cc:5582] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 31 to 32, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New cstate: current_term: 4 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:16.000659 3828 raft_consensus.cc:2953] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:16.003743 3890 tablet_service.cc:1515] Processing DeleteTablet for tablet b045aeb5889a4b88b8ee72370f2a7e39 with delete_type TABLET_DATA_TOMBSTONED (TS 3377fd0584c546c1a938e3a1204b2be3 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:36332
I20250622 13:53:16.004170 4733 tablet_replica.cc:331] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:16.004340 4733 raft_consensus.cc:2241] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:16.004617 4733 raft_consensus.cc:2270] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:16.005808 4733 ts_tablet_manager.cc:1905] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:16.011476 4733 ts_tablet_manager.cc:1918] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.31
I20250622 13:53:16.011616 4733 log.cc:1199] T b045aeb5889a4b88b8ee72370f2a7e39 P 3377fd0584c546c1a938e3a1204b2be3: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root/wals/b045aeb5889a4b88b8ee72370f2a7e39
I20250622 13:53:16.012225 3643 catalog_manager.cc:4928] TS 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): tablet b045aeb5889a4b88b8ee72370f2a7e39 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:16.100731 4662 raft_consensus.cc:1062] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: attempting to promote NON_VOTER 52c21167280a4d669123bb1ed7675d79 to VOTER
I20250622 13:53:16.101449 4662 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 32, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:16.103554 4332 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 LEARNER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20250622 13:53:16.103554 4184 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20250622 13:53:16.104120 4745 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 13:53:16.104390 3828 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20250622 13:53:16.104458 4662 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 13:53:16.105125 4662 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 13:53:16.112828 4745 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Committing config change with OpId 4.35: config changed from index 34 to 35, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:16.113446 4332 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:16.113682 4184 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:16.116024 3828 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:16.116801 3656 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 34 to 35, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:16.123699 3903 consensus_queue.cc:237] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.35, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:16.125326 4332 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20250622 13:53:16.125445 4184 raft_consensus.cc:1273] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20250622 13:53:16.126041 4734 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250622 13:53:16.126482 4662 consensus_queue.cc:1035] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250622 13:53:16.128604 4745 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Committing config change with OpId 4.36: config changed from index 35 to 36, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:16.129037 4183 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:16.128970 4332 raft_consensus.cc:2953] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:16.133538 3655 catalog_manager.cc:5582] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 reported cstate change: config changed from index 35 to 36, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New cstate: current_term: 4 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:16.134392 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c9260a075f104769bc81ca7eb0c76cb2 with cas_config_opid_index 35: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:16.137063 3819 tablet_service.cc:1515] Processing DeleteTablet for tablet c9260a075f104769bc81ca7eb0c76cb2 with delete_type TABLET_DATA_TOMBSTONED (TS 946d416d6f4a4b0ea6a77b93cd6f58f6 not found in new config with opid_index 36) from {username='slave'} at 127.0.0.1:45604
I20250622 13:53:16.137758 4750 tablet_replica.cc:331] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:16.137985 4750 raft_consensus.cc:2241] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:16.138255 4750 raft_consensus.cc:2270] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:16.139142 4750 ts_tablet_manager.cc:1905] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:16.143138 4750 ts_tablet_manager.cc:1918] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.35
I20250622 13:53:16.143246 4750 log.cc:1199] T c9260a075f104769bc81ca7eb0c76cb2 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root/wals/c9260a075f104769bc81ca7eb0c76cb2
I20250622 13:53:16.143814 3643 catalog_manager.cc:4928] TS 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): tablet c9260a075f104769bc81ca7eb0c76cb2 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:16.262599 3903 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:16.263232 4737 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:16.266675 4259 raft_consensus.cc:1273] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:16.267279 4737 consensus_queue.cc:1035] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:17.141575 3903 consensus_queue.cc:237] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:17.143884 3756 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
W20250622 13:53:17.144011 3874 consensus_peers.cc:487] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 -> Peer feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925): Couldn't send request to peer feaebd1ce3104a2fa3cc207fd3c23468. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3c17332dc1c4acaa727cf9b2fb8ed8e. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:17.144098 3828 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20250622 13:53:17.144407 4745 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:17.144672 4759 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:17.147238 4759 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 5 LEADER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:17.147635 3756 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:17.148029 3828 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:17.152935 3651 catalog_manager.cc:5582] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index -1 to 31, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New cstate: current_term: 5 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:17.153748 4183 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:17.156447 3756 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250622 13:53:17.157172 4766 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:17.157608 3903 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250622 13:53:17.158351 4766 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:17.161769 4699 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 30 to 33, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:17.162446 3903 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 30 to 33, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:17.166273 3828 consensus_queue.cc:237] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:17.166859 3656 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 30 to 33, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New cstate: current_term: 3 leader_uuid: "d23b47356efa46c792ab125b568ea95a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:17.175136 3753 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20250622 13:53:17.175285 3797 consensus_peers.cc:487] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 -> Peer d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811): Couldn't send request to peer d23b47356efa46c792ab125b568ea95a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fd3fa41ece64410b7448f59d5db9ba7. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:17.176378 4706 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:17.176482 3753 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 30 to 33, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:17.178813 3903 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250622 13:53:17.179524 4706 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20250622 13:53:17.184453 4148 consensus_peers.cc:487] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a -> Peer 52c21167280a4d669123bb1ed7675d79 (127.0.103.70:35833): Couldn't send request to peer 52c21167280a4d669123bb1ed7675d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5246e7fae8d44ba6a5e3ac826f9988f4. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:17.196069 4777 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:17.198068 3756 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:17.210327 3908 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:17.214645 3656 catalog_manager.cc:5582] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index -1 to 30, NON_VOTER d23b47356efa46c792ab125b568ea95a (127.0.103.68) added. New cstate: current_term: 3 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } } }
I20250622 13:53:17.215605 3756 consensus_queue.cc:237] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:17.218011 3828 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250622 13:53:17.218545 3908 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250622 13:53:17.219089 4737 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:17.219825 4737 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:17.223135 4757 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:17.224115 3828 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:17.229068 3828 consensus_queue.cc:237] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:17.229255 3655 catalog_manager.cc:5582] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index -1 to 30, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New cstate: current_term: 3 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:17.229565 3908 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } } }
I20250622 13:53:17.232272 3756 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:53:17.232831 4778 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20250622 13:53:17.233539 3903 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250622 13:53:17.234573 4777 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20250622 13:53:17.238372 3723 consensus_peers.cc:487] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 -> Peer 52c21167280a4d669123bb1ed7675d79 (127.0.103.70:35833): Couldn't send request to peer 52c21167280a4d669123bb1ed7675d79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bac2340a96e94c219f59c12945eb8bf6. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:17.239861 4706 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:17.240470 3908 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:17.245994 3651 catalog_manager.cc:5582] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
W20250622 13:53:17.256260 3800 consensus_peers.cc:487] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 -> Peer feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925): Couldn't send request to peer feaebd1ce3104a2fa3cc207fd3c23468. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5c429ec7a7a4a7c8b9b50ae4be5ff59. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:17.257493 3756 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } } }
I20250622 13:53:17.586323 4789 ts_tablet_manager.cc:927] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: Initiating tablet copy from peer 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:17.586944 4789 tablet_copy_client.cc:323] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33649
I20250622 13:53:17.587463 3918 tablet_copy_service.cc:140] P 3377fd0584c546c1a938e3a1204b2be3: Received BeginTabletCopySession request for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e from peer feaebd1ce3104a2fa3cc207fd3c23468 ({username='slave'} at 127.0.0.1:36388)
I20250622 13:53:17.587584 3918 tablet_copy_service.cc:161] P 3377fd0584c546c1a938e3a1204b2be3: Beginning new tablet copy session on tablet c3c17332dc1c4acaa727cf9b2fb8ed8e from peer feaebd1ce3104a2fa3cc207fd3c23468 at {username='slave'} at 127.0.0.1:36388: session id = feaebd1ce3104a2fa3cc207fd3c23468-c3c17332dc1c4acaa727cf9b2fb8ed8e
I20250622 13:53:17.589715 3918 tablet_copy_source_session.cc:215] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:17.590659 4789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:17.595425 4789 tablet_copy_client.cc:806] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:17.595739 4789 tablet_copy_client.cc:670] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:17.597069 4790 tablet_copy_client.cc:1025] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:17.637209 4791 ts_tablet_manager.cc:927] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: Initiating tablet copy from peer d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517)
I20250622 13:53:17.637861 4791 tablet_copy_client.cc:323] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:40517
I20250622 13:53:17.638418 3769 tablet_copy_service.cc:140] P d13e1e991fdc47899aac5b7c33fbbbd9: Received BeginTabletCopySession request for tablet bac2340a96e94c219f59c12945eb8bf6 from peer 52c21167280a4d669123bb1ed7675d79 ({username='slave'} at 127.0.0.1:53860)
I20250622 13:53:17.638557 3769 tablet_copy_service.cc:161] P d13e1e991fdc47899aac5b7c33fbbbd9: Beginning new tablet copy session on tablet bac2340a96e94c219f59c12945eb8bf6 from peer 52c21167280a4d669123bb1ed7675d79 at {username='slave'} at 127.0.0.1:53860: session id = 52c21167280a4d669123bb1ed7675d79-bac2340a96e94c219f59c12945eb8bf6
I20250622 13:53:17.640949 3769 tablet_copy_source_session.cc:215] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:17.642055 4791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac2340a96e94c219f59c12945eb8bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:17.647053 4791 tablet_copy_client.cc:806] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:17.647289 4791 tablet_copy_client.cc:670] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:17.648598 4792 tablet_copy_client.cc:1025] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:17.700670 4793 ts_tablet_manager.cc:927] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: Initiating tablet copy from peer d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
I20250622 13:53:17.701256 4793 tablet_copy_client.cc:323] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:33811
I20250622 13:53:17.708236 4194 tablet_copy_service.cc:140] P d23b47356efa46c792ab125b568ea95a: Received BeginTabletCopySession request for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 from peer 52c21167280a4d669123bb1ed7675d79 ({username='slave'} at 127.0.0.1:58124)
I20250622 13:53:17.708362 4194 tablet_copy_service.cc:161] P d23b47356efa46c792ab125b568ea95a: Beginning new tablet copy session on tablet 5246e7fae8d44ba6a5e3ac826f9988f4 from peer 52c21167280a4d669123bb1ed7675d79 at {username='slave'} at 127.0.0.1:58124: session id = 52c21167280a4d669123bb1ed7675d79-5246e7fae8d44ba6a5e3ac826f9988f4
I20250622 13:53:17.710567 4194 tablet_copy_source_session.cc:215] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:17.711764 4793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5246e7fae8d44ba6a5e3ac826f9988f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:17.716514 4793 tablet_copy_client.cc:806] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:17.716728 4793 tablet_copy_client.cc:670] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:17.718212 4794 tablet_copy_client.cc:1025] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:17.760183 4797 ts_tablet_manager.cc:927] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: Initiating tablet copy from peer 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:53:17.760738 4797 tablet_copy_client.cc:323] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:34391
I20250622 13:53:17.761277 3844 tablet_copy_service.cc:140] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Received BeginTabletCopySession request for tablet 6fd3fa41ece64410b7448f59d5db9ba7 from peer d23b47356efa46c792ab125b568ea95a ({username='slave'} at 127.0.0.1:45644)
I20250622 13:53:17.761443 3844 tablet_copy_service.cc:161] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Beginning new tablet copy session on tablet 6fd3fa41ece64410b7448f59d5db9ba7 from peer d23b47356efa46c792ab125b568ea95a at {username='slave'} at 127.0.0.1:45644: session id = d23b47356efa46c792ab125b568ea95a-6fd3fa41ece64410b7448f59d5db9ba7
I20250622 13:53:17.763927 3844 tablet_copy_source_session.cc:215] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:17.764936 4797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd3fa41ece64410b7448f59d5db9ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:17.770390 4797 tablet_copy_client.cc:806] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:17.770536 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:17.770598 4797 tablet_copy_client.cc:670] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:17.772254 4798 tablet_copy_client.cc:1025] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:17.773840 3756 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:17.774017 3756 raft_consensus.cc:604] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Received request to transfer leadership to TS 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:53:17.774439 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:17.774626 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:17.779124 4799 ts_tablet_manager.cc:927] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: Initiating tablet copy from peer 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391)
I20250622 13:53:17.779687 4799 tablet_copy_client.cc:323] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:34391
I20250622 13:53:17.780228 3844 tablet_copy_service.cc:140] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Received BeginTabletCopySession request for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 from peer feaebd1ce3104a2fa3cc207fd3c23468 ({username='slave'} at 127.0.0.1:45670)
I20250622 13:53:17.780364 3844 tablet_copy_service.cc:161] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Beginning new tablet copy session on tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 from peer feaebd1ce3104a2fa3cc207fd3c23468 at {username='slave'} at 127.0.0.1:45670: session id = feaebd1ce3104a2fa3cc207fd3c23468-b5c429ec7a7a4a7c8b9b50ae4be5ff59
I20250622 13:53:17.782549 3844 tablet_copy_source_session.cc:215] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:17.783576 4799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:17.789124 4799 tablet_copy_client.cc:806] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:17.789336 4799 tablet_copy_client.cc:670] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:17.790586 4800 tablet_copy_client.cc:1025] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Injecting latency into file download: 2000
I20250622 13:53:18.042932 4757 raft_consensus.cc:991] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:18.043105 4757 raft_consensus.cc:1079] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Signalling peer 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:18.044055 3828 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:45606
I20250622 13:53:18.044255 3828 raft_consensus.cc:491] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:18.044353 3828 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:18.046757 3828 raft_consensus.cc:513] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:18.047544 3828 leader_election.cc:290] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 5 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
I20250622 13:53:18.047676 3756 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:18.047727 4184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "d23b47356efa46c792ab125b568ea95a"
I20250622 13:53:18.047879 4184 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:18.047879 3756 raft_consensus.cc:3053] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Stepping down as leader of term 4
I20250622 13:53:18.048143 3756 raft_consensus.cc:738] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:18.048405 3756 consensus_queue.cc:260] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:18.048725 3756 raft_consensus.cc:3058] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:18.049981 4184 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 5.
I20250622 13:53:18.050478 3797 leader_election.cc:304] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d23b47356efa46c792ab125b568ea95a; no voters:
I20250622 13:53:18.050659 3756 raft_consensus.cc:2466] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 5.
I20250622 13:53:18.050794 4706 raft_consensus.cc:2802] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Leader election won for term 5
I20250622 13:53:18.051229 4706 raft_consensus.cc:695] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:18.051528 4706 consensus_queue.cc:237] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:18.054059 3655 catalog_manager.cc:5582] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 4 to 5, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 5 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:18.491978 3756 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250622 13:53:18.492486 4812 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:18.496412 4184 raft_consensus.cc:1273] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 5 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250622 13:53:18.496994 4706 consensus_queue.cc:1035] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:19.597405 4789 tablet_copy_client.cc:538] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:19.600462 4789 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap starting.
I20250622 13:53:19.648943 4791 tablet_copy_client.cc:538] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:19.652086 4791 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: Bootstrap starting.
I20250622 13:53:19.657714 4789 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:19.658066 4789 tablet_bootstrap.cc:492] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap complete.
I20250622 13:53:19.658320 4789 ts_tablet_manager.cc:1397] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: Time spent bootstrapping tablet: real 0.058s user 0.059s sys 0.001s
I20250622 13:53:19.659049 4789 raft_consensus.cc:357] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:19.659204 4789 raft_consensus.cc:738] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: feaebd1ce3104a2fa3cc207fd3c23468, State: Initialized, Role: LEARNER
I20250622 13:53:19.659405 4789 consensus_queue.cc:260] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:19.659930 4789 ts_tablet_manager.cc:1428] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:53:19.660609 3918 tablet_copy_service.cc:342] P 3377fd0584c546c1a938e3a1204b2be3: Request end of tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-c3c17332dc1c4acaa727cf9b2fb8ed8e received from {username='slave'} at 127.0.0.1:36388
I20250622 13:53:19.660708 3918 tablet_copy_service.cc:434] P 3377fd0584c546c1a938e3a1204b2be3: ending tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-c3c17332dc1c4acaa727cf9b2fb8ed8e on tablet c3c17332dc1c4acaa727cf9b2fb8ed8e with peer feaebd1ce3104a2fa3cc207fd3c23468
I20250622 13:53:19.693089 4791 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=91 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:19.693583 4791 tablet_bootstrap.cc:492] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: Bootstrap complete.
I20250622 13:53:19.693831 4791 ts_tablet_manager.cc:1397] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: Time spent bootstrapping tablet: real 0.042s user 0.036s sys 0.007s
I20250622 13:53:19.694623 4791 raft_consensus.cc:357] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:19.694828 4791 raft_consensus.cc:738] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 52c21167280a4d669123bb1ed7675d79, State: Initialized, Role: LEARNER
I20250622 13:53:19.695011 4791 consensus_queue.cc:260] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:19.695571 4791 ts_tablet_manager.cc:1428] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:53:19.696103 3769 tablet_copy_service.cc:342] P d13e1e991fdc47899aac5b7c33fbbbd9: Request end of tablet copy session 52c21167280a4d669123bb1ed7675d79-bac2340a96e94c219f59c12945eb8bf6 received from {username='slave'} at 127.0.0.1:53860
I20250622 13:53:19.696215 3769 tablet_copy_service.cc:434] P d13e1e991fdc47899aac5b7c33fbbbd9: ending tablet copy session 52c21167280a4d669123bb1ed7675d79-bac2340a96e94c219f59c12945eb8bf6 on tablet bac2340a96e94c219f59c12945eb8bf6 with peer 52c21167280a4d669123bb1ed7675d79
I20250622 13:53:19.718565 4793 tablet_copy_client.cc:538] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:19.721592 4793 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: Bootstrap starting.
I20250622 13:53:19.760830 4793 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:19.761281 4793 tablet_bootstrap.cc:492] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: Bootstrap complete.
I20250622 13:53:19.761579 4793 ts_tablet_manager.cc:1397] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: Time spent bootstrapping tablet: real 0.040s user 0.039s sys 0.000s
I20250622 13:53:19.762344 4793 raft_consensus.cc:357] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:19.762501 4793 raft_consensus.cc:738] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 52c21167280a4d669123bb1ed7675d79, State: Initialized, Role: LEARNER
I20250622 13:53:19.762643 4793 consensus_queue.cc:260] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: true } }
I20250622 13:53:19.763105 4793 ts_tablet_manager.cc:1428] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250622 13:53:19.763816 4194 tablet_copy_service.cc:342] P d23b47356efa46c792ab125b568ea95a: Request end of tablet copy session 52c21167280a4d669123bb1ed7675d79-5246e7fae8d44ba6a5e3ac826f9988f4 received from {username='slave'} at 127.0.0.1:58124
I20250622 13:53:19.763932 4194 tablet_copy_service.cc:434] P d23b47356efa46c792ab125b568ea95a: ending tablet copy session 52c21167280a4d669123bb1ed7675d79-5246e7fae8d44ba6a5e3ac826f9988f4 on tablet 5246e7fae8d44ba6a5e3ac826f9988f4 with peer 52c21167280a4d669123bb1ed7675d79
I20250622 13:53:19.772604 4797 tablet_copy_client.cc:538] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:19.774915 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:19.775518 4797 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: Bootstrap starting.
I20250622 13:53:19.778218 3832 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:19.778409 3832 raft_consensus.cc:604] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:53:19.778815 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:19.779057 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:19.790935 4799 tablet_copy_client.cc:538] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:19.794335 4799 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap starting.
I20250622 13:53:19.810387 4797 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:19.810779 4797 tablet_bootstrap.cc:492] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: Bootstrap complete.
I20250622 13:53:19.811033 4797 ts_tablet_manager.cc:1397] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: Time spent bootstrapping tablet: real 0.036s user 0.032s sys 0.004s
I20250622 13:53:19.811698 4797 raft_consensus.cc:357] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:19.812026 4797 raft_consensus.cc:738] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: d23b47356efa46c792ab125b568ea95a, State: Initialized, Role: LEARNER
I20250622 13:53:19.812215 4797 consensus_queue.cc:260] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: true } }
I20250622 13:53:19.813102 4797 ts_tablet_manager.cc:1428] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:19.814899 3844 tablet_copy_service.cc:342] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Request end of tablet copy session d23b47356efa46c792ab125b568ea95a-6fd3fa41ece64410b7448f59d5db9ba7 received from {username='slave'} at 127.0.0.1:45644
I20250622 13:53:19.815001 3844 tablet_copy_service.cc:434] P 946d416d6f4a4b0ea6a77b93cd6f58f6: ending tablet copy session d23b47356efa46c792ab125b568ea95a-6fd3fa41ece64410b7448f59d5db9ba7 on tablet 6fd3fa41ece64410b7448f59d5db9ba7 with peer d23b47356efa46c792ab125b568ea95a
I20250622 13:53:19.831856 4799 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:19.832221 4799 tablet_bootstrap.cc:492] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: Bootstrap complete.
I20250622 13:53:19.832473 4799 ts_tablet_manager.cc:1397] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: Time spent bootstrapping tablet: real 0.038s user 0.030s sys 0.005s
I20250622 13:53:19.833235 4799 raft_consensus.cc:357] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:19.833559 4799 raft_consensus.cc:738] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: feaebd1ce3104a2fa3cc207fd3c23468, State: Initialized, Role: LEARNER
I20250622 13:53:19.833786 4799 consensus_queue.cc:260] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:19.834239 4799 ts_tablet_manager.cc:1428] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250622 13:53:19.835036 3844 tablet_copy_service.cc:342] P 946d416d6f4a4b0ea6a77b93cd6f58f6: Request end of tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-b5c429ec7a7a4a7c8b9b50ae4be5ff59 received from {username='slave'} at 127.0.0.1:45670
I20250622 13:53:19.835139 3844 tablet_copy_service.cc:434] P 946d416d6f4a4b0ea6a77b93cd6f58f6: ending tablet copy session feaebd1ce3104a2fa3cc207fd3c23468-b5c429ec7a7a4a7c8b9b50ae4be5ff59 on tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 with peer feaebd1ce3104a2fa3cc207fd3c23468
I20250622 13:53:19.838693 4812 raft_consensus.cc:991] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:19.838819 4812 raft_consensus.cc:1079] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Signalling peer d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:19.839335 3756 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:53780
I20250622 13:53:19.839468 3756 raft_consensus.cc:491] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:19.839536 3756 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:19.841611 3756 raft_consensus.cc:513] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:19.842160 3756 leader_election.cc:290] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 2 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649)
I20250622 13:53:19.842474 3832 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:19.842617 3832 raft_consensus.cc:3053] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:19.842681 3832 raft_consensus.cc:738] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:19.842921 3832 consensus_queue.cc:260] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:19.843307 3832 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:19.845189 3832 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 2.
I20250622 13:53:19.845543 3908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:19.845582 3724 leader_election.cc:304] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d13e1e991fdc47899aac5b7c33fbbbd9; no voters:
I20250622 13:53:19.845690 3908 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:19.845845 4823 raft_consensus.cc:2802] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:19.846171 4823 raft_consensus.cc:695] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:19.846434 4823 consensus_queue.cc:237] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } }
I20250622 13:53:19.848201 3908 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 2.
I20250622 13:53:19.849143 3651 catalog_manager.cc:5582] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 1 to 2, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:19.920940 4184 raft_consensus.cc:1215] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 3 LEARNER]: Deduplicated request from leader. Original: 3.29->[3.30-3.30] Dedup: 3.30->[]
I20250622 13:53:19.983575 4259 raft_consensus.cc:1215] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.30->[5.31-5.31] Dedup: 5.31->[]
I20250622 13:53:20.009976 4332 raft_consensus.cc:1215] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.32->[3.33-3.33] Dedup: 3.33->[]
I20250622 13:53:20.225992 4332 raft_consensus.cc:1215] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.29->[3.30-3.30] Dedup: 3.30->[]
I20250622 13:53:20.321174 3832 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250622 13:53:20.321707 4823 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:20.334767 4259 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 1 LEARNER]: Advancing to term 2
I20250622 13:53:20.335009 3908 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250622 13:53:20.335505 4841 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:20.337905 4259 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEARNER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250622 13:53:20.339349 4830 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250622 13:53:20.342996 4830 raft_consensus.cc:1062] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: attempting to promote NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 to VOTER
I20250622 13:53:20.343792 4830 consensus_queue.cc:237] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:20.345657 3832 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250622 13:53:20.345760 3908 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250622 13:53:20.346097 4259 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEARNER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250622 13:53:20.346202 4823 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:20.346402 4841 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:20.346539 4830 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250622 13:53:20.348950 4823 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.350739 4259 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.353253 3908 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.353713 3832 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.354257 3651 catalog_manager.cc:5582] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index 28 to 30, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:20.372735 3753 consensus_queue.cc:237] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:20.374383 4259 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:20.375042 4841 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:20.375872 3832 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250622 13:53:20.376508 4841 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:20.377511 4830 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.381568 3644 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:20.384171 3652 catalog_manager.cc:5582] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index 30 to 31, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:20.385099 4849 raft_consensus.cc:1062] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: attempting to promote NON_VOTER d23b47356efa46c792ab125b568ea95a to VOTER
I20250622 13:53:20.383339 3832 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.385952 4849 consensus_queue.cc:237] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:20.387883 3890 tablet_service.cc:1515] Processing DeleteTablet for tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 with delete_type TABLET_DATA_TOMBSTONED (TS 3377fd0584c546c1a938e3a1204b2be3 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:36332
I20250622 13:53:20.388463 4852 tablet_replica.cc:331] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:20.388636 4852 raft_consensus.cc:2241] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:20.388917 4852 raft_consensus.cc:2270] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:20.391170 4852 ts_tablet_manager.cc:1905] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:20.391381 4259 raft_consensus.cc:2953] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.392688 3908 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250622 13:53:20.393630 4184 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 3 LEARNER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250622 13:53:20.398913 4852 ts_tablet_manager.cc:1918] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20250622 13:53:20.399039 4852 log.cc:1199] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 3377fd0584c546c1a938e3a1204b2be3: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-2-root/wals/b5c429ec7a7a4a7c8b9b50ae4be5ff59
I20250622 13:53:20.399638 3643 catalog_manager.cc:4928] TS 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649): tablet b5c429ec7a7a4a7c8b9b50ae4be5ff59 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:20.400614 4812 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:20.401305 4812 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:20.408854 3753 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250622 13:53:20.409438 4850 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:20.412809 4849 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:20.413482 3756 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:20.417811 3652 catalog_manager.cc:5582] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:20.421386 4184 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:20.432507 3908 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, d23b47356efa46c792ab125b568ea95a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:20.434803 3832 consensus_queue.cc:237] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:20.436679 3902 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:20.436761 4184 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:20.437168 4853 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:20.437588 4853 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:20.439405 4853 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:20.440203 4184 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:20.440124 3902 raft_consensus.cc:2953] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } }
I20250622 13:53:20.443154 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6fd3fa41ece64410b7448f59d5db9ba7 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:20.444247 3652 catalog_manager.cc:5582] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: config changed from index 31 to 32, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New cstate: current_term: 3 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:20.448499 3745 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fd3fa41ece64410b7448f59d5db9ba7 with delete_type TABLET_DATA_TOMBSTONED (TS d13e1e991fdc47899aac5b7c33fbbbd9 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:53772
I20250622 13:53:20.449206 4857 tablet_replica.cc:331] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:20.449450 4857 raft_consensus.cc:2241] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:20.449739 4857 raft_consensus.cc:2270] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:20.450577 4857 ts_tablet_manager.cc:1905] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:20.455142 4857 ts_tablet_manager.cc:1918] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20250622 13:53:20.455319 4857 log.cc:1199] T 6fd3fa41ece64410b7448f59d5db9ba7 P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root/wals/6fd3fa41ece64410b7448f59d5db9ba7
I20250622 13:53:20.455974 3644 catalog_manager.cc:4928] TS d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517): tablet 6fd3fa41ece64410b7448f59d5db9ba7 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:20.512890 4859 raft_consensus.cc:1062] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: attempting to promote NON_VOTER 52c21167280a4d669123bb1ed7675d79 to VOTER
I20250622 13:53:20.513619 4859 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 31, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:20.515626 3908 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250622 13:53:20.515735 4332 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 LEARNER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250622 13:53:20.516060 4858 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:20.516317 4836 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:20.516906 3756 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250622 13:53:20.519011 4836 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:20.520328 4836 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.525566 4332 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.526387 3753 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.526932 3652 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 33 to 34, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d23b47356efa46c792ab125b568ea95a" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:20.525568 3908 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.533486 4183 consensus_queue.cc:237] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:20.535223 4332 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20250622 13:53:20.535374 3908 raft_consensus.cc:1273] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d23b47356efa46c792ab125b568ea95a: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20250622 13:53:20.535651 4865 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 13:53:20.536134 4861 consensus_queue.cc:1035] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250622 13:53:20.538425 4858 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 3 LEADER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.539078 4332 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.541549 3699 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:20.542429 3908 raft_consensus.cc:2953] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.542905 3651 catalog_manager.cc:5582] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a reported cstate change: config changed from index 34 to 35, VOTER d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) evicted. New cstate: current_term: 3 leader_uuid: "d23b47356efa46c792ab125b568ea95a" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:20.546809 3745 tablet_service.cc:1515] Processing DeleteTablet for tablet 5246e7fae8d44ba6a5e3ac826f9988f4 with delete_type TABLET_DATA_TOMBSTONED (TS d13e1e991fdc47899aac5b7c33fbbbd9 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:53772
I20250622 13:53:20.547147 4857 tablet_replica.cc:331] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:20.547329 4857 raft_consensus.cc:2241] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:20.547573 4857 raft_consensus.cc:2270] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:20.548408 4857 ts_tablet_manager.cc:1905] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:20.553680 4857 ts_tablet_manager.cc:1918] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.34
I20250622 13:53:20.553808 4857 log.cc:1199] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d13e1e991fdc47899aac5b7c33fbbbd9: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-0-root/wals/5246e7fae8d44ba6a5e3ac826f9988f4
I20250622 13:53:20.554570 3644 catalog_manager.cc:4928] TS d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517): tablet 5246e7fae8d44ba6a5e3ac826f9988f4 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:20.600205 4867 raft_consensus.cc:1062] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: attempting to promote NON_VOTER feaebd1ce3104a2fa3cc207fd3c23468 to VOTER
I20250622 13:53:20.600934 4867 consensus_queue.cc:237] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 29, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:20.603108 4259 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 LEARNER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250622 13:53:20.603368 3832 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250622 13:53:20.603628 4869 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:20.603569 3753 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250622 13:53:20.603950 4819 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:20.604311 4867 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:20.606992 4869 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 5 LEADER]: Committing config change with OpId 5.32: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.607717 4259 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.610950 3832 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.615749 3651 catalog_manager.cc:5582] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:20.621456 3753 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.623675 3902 consensus_queue.cc:237] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:20.625627 3756 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250622 13:53:20.625856 4259 raft_consensus.cc:1273] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250622 13:53:20.626127 4869 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:20.627106 4869 consensus_queue.cc:1035] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:20.628087 4819 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.629873 3753 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.630470 4259 raft_consensus.cc:2953] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } } }
I20250622 13:53:20.631085 3643 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:20.632265 3652 catalog_manager.cc:5582] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: config changed from index 32 to 33, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New cstate: current_term: 5 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:20.636194 3819 tablet_service.cc:1515] Processing DeleteTablet for tablet c3c17332dc1c4acaa727cf9b2fb8ed8e with delete_type TABLET_DATA_TOMBSTONED (TS 946d416d6f4a4b0ea6a77b93cd6f58f6 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:45604
I20250622 13:53:20.637027 4873 tablet_replica.cc:331] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:20.637219 4873 raft_consensus.cc:2241] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:20.637491 4873 raft_consensus.cc:2270] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:20.638769 4873 ts_tablet_manager.cc:1905] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:20.644193 4873 ts_tablet_manager.cc:1918] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.32
I20250622 13:53:20.644328 4873 log.cc:1199] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root/wals/c3c17332dc1c4acaa727cf9b2fb8ed8e
I20250622 13:53:20.645025 3643 catalog_manager.cc:4928] TS 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): tablet c3c17332dc1c4acaa727cf9b2fb8ed8e (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:20.718506 4830 raft_consensus.cc:1062] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: attempting to promote NON_VOTER 52c21167280a4d669123bb1ed7675d79 to VOTER
I20250622 13:53:20.719430 4830 consensus_queue.cc:237] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:20.721508 4332 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 LEARNER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250622 13:53:20.721508 3902 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250622 13:53:20.721820 3828 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250622 13:53:20.722272 4830 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:20.722534 4841 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:20.723059 4823 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250622 13:53:20.726990 4840 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.727538 4332 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.730234 3828 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.730329 3902 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.732127 3655 catalog_manager.cc:5582] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 reported cstate change: config changed from index 30 to 31, 52c21167280a4d669123bb1ed7675d79 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } attrs { replace: true } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.738178 3756 consensus_queue.cc:237] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } }
I20250622 13:53:20.739670 4332 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:20.740053 3902 raft_consensus.cc:1273] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:20.740129 4830 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:20.741232 4830 consensus_queue.cc:1035] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:20.742215 4840 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.743182 4332 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.745054 3644 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bac2340a96e94c219f59c12945eb8bf6 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:20.746429 3651 catalog_manager.cc:5582] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: config changed from index 31 to 32, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New cstate: current_term: 3 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:53:20.747864 3902 raft_consensus.cc:2953] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "52c21167280a4d669123bb1ed7675d79" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 35833 } attrs { promote: false } } }
I20250622 13:53:20.750051 3819 tablet_service.cc:1515] Processing DeleteTablet for tablet bac2340a96e94c219f59c12945eb8bf6 with delete_type TABLET_DATA_TOMBSTONED (TS 946d416d6f4a4b0ea6a77b93cd6f58f6 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:45604
I20250622 13:53:20.750336 4873 tablet_replica.cc:331] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:20.750514 4873 raft_consensus.cc:2241] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:20.750758 4873 raft_consensus.cc:2270] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:20.751749 4873 ts_tablet_manager.cc:1905] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:20.755836 4873 ts_tablet_manager.cc:1918] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20250622 13:53:20.755942 4873 log.cc:1199] T bac2340a96e94c219f59c12945eb8bf6 P 946d416d6f4a4b0ea6a77b93cd6f58f6: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0/minicluster-data/ts-1-root/wals/bac2340a96e94c219f59c12945eb8bf6
I20250622 13:53:20.756517 3643 catalog_manager.cc:4928] TS 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391): tablet bac2340a96e94c219f59c12945eb8bf6 (table test-workload [id=87e5125c90424c10bc2da4f561da8175]) successfully deleted
I20250622 13:53:21.779465 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:21.782063 3753 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:21.782235 3753 raft_consensus.cc:604] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Received request to transfer leadership to TS 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:53:21.782697 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:21.782860 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:22.071396 4840 raft_consensus.cc:991] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:22.071547 4840 raft_consensus.cc:1079] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Signalling peer 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:22.072103 3828 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:45606
I20250622 13:53:22.072278 3828 raft_consensus.cc:491] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:22.072398 3828 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:22.074268 3828 raft_consensus.cc:513] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:22.074815 3828 leader_election.cc:290] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 3 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925)
I20250622 13:53:22.075246 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:22.075399 3753 raft_consensus.cc:3053] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:53:22.075493 3753 raft_consensus.cc:738] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:22.075404 4259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
I20250622 13:53:22.075670 4259 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:22.075749 3753 consensus_queue.cc:260] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:22.076021 3753 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:22.077770 4259 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 3.
I20250622 13:53:22.077884 3753 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 3.
I20250622 13:53:22.078212 3800 leader_election.cc:304] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, feaebd1ce3104a2fa3cc207fd3c23468; no voters:
I20250622 13:53:22.078439 4812 raft_consensus.cc:2802] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:53:22.078758 4812 raft_consensus.cc:695] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:22.079011 4812 consensus_queue.cc:237] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:22.081053 3652 catalog_manager.cc:5582] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 2 to 3, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 3 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:22.456218 3753 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:22.456779 4812 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:22.460942 4259 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 3 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250622 13:53:22.461856 4812 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250622 13:53:23.783232 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:23.785957 3828 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:23.786186 3828 raft_consensus.cc:604] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Received request to transfer leadership to TS 3377fd0584c546c1a938e3a1204b2be3
I20250622 13:53:23.786681 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:23.786912 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:23.891714 4812 raft_consensus.cc:991] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:23.891868 4812 raft_consensus.cc:1079] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Signalling peer 3377fd0584c546c1a938e3a1204b2be3 to start an election
I20250622 13:53:23.892467 3902 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
from {username='slave'} at 127.0.0.1:36344
I20250622 13:53:23.892648 3902 raft_consensus.cc:491] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:23.892738 3902 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:23.894660 3902 raft_consensus.cc:513] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:23.895304 3902 leader_election.cc:290] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 4 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
I20250622 13:53:23.895674 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:23.895717 4183 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "3377fd0584c546c1a938e3a1204b2be3" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "d23b47356efa46c792ab125b568ea95a"
I20250622 13:53:23.895840 3828 raft_consensus.cc:3053] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Stepping down as leader of term 3
I20250622 13:53:23.895892 4183 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:23.895936 3828 raft_consensus.cc:738] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:23.896229 3828 consensus_queue.cc:260] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:23.896497 3828 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:23.897809 4183 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 4.
I20250622 13:53:23.898229 3871 leader_election.cc:304] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3377fd0584c546c1a938e3a1204b2be3, d23b47356efa46c792ab125b568ea95a; no voters:
I20250622 13:53:23.898397 3828 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3377fd0584c546c1a938e3a1204b2be3 in term 4.
I20250622 13:53:23.898505 4867 raft_consensus.cc:2802] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 13:53:23.898658 4867 raft_consensus.cc:695] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Becoming Leader. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:23.898968 4867 consensus_queue.cc:237] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:23.901207 3651 catalog_manager.cc:5582] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 reported cstate change: term changed from 3 to 4, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67). New cstate: current_term: 4 leader_uuid: "3377fd0584c546c1a938e3a1204b2be3" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:24.355265 3828 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250622 13:53:24.356027 4867 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:24.359689 4183 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Refusing update from remote peer 3377fd0584c546c1a938e3a1204b2be3: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250622 13:53:24.360219 4867 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:25.787308 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:25.806440 3902 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7"
dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
mode: GRACEFUL
new_leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:36364
I20250622 13:53:25.806648 3902 raft_consensus.cc:604] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Received request to transfer leadership to TS 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:53:25.807075 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:25.807253 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:25.906417 4893 raft_consensus.cc:991] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3: : Instructing follower 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:25.906543 4893 raft_consensus.cc:1079] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Signalling peer 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:25.907104 3828 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:45620
I20250622 13:53:25.907291 3828 raft_consensus.cc:491] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:25.907382 3828 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:25.909134 3828 raft_consensus.cc:513] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:25.909732 3828 leader_election.cc:290] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 5 election: Requested vote from peers 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67:33649), d23b47356efa46c792ab125b568ea95a (127.0.103.68:33811)
I20250622 13:53:25.910203 4183 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "d23b47356efa46c792ab125b568ea95a"
I20250622 13:53:25.910334 3902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fd3fa41ece64410b7448f59d5db9ba7" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "3377fd0584c546c1a938e3a1204b2be3"
I20250622 13:53:25.910357 4183 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:25.910526 3902 raft_consensus.cc:3053] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Stepping down as leader of term 4
I20250622 13:53:25.910588 3902 raft_consensus.cc:738] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3377fd0584c546c1a938e3a1204b2be3, State: Running, Role: LEADER
I20250622 13:53:25.910838 3902 consensus_queue.cc:260] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:25.911149 3902 raft_consensus.cc:3058] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:25.913234 4183 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 5.
I20250622 13:53:25.913611 3902 raft_consensus.cc:2466] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 5.
I20250622 13:53:25.913694 3797 leader_election.cc:304] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, d23b47356efa46c792ab125b568ea95a; no voters:
I20250622 13:53:25.913981 4812 raft_consensus.cc:2802] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Leader election won for term 5
I20250622 13:53:25.914302 4812 raft_consensus.cc:695] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:25.914556 4812 consensus_queue.cc:237] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } }
I20250622 13:53:25.916671 3652 catalog_manager.cc:5582] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 4 to 5, leader changed from 3377fd0584c546c1a938e3a1204b2be3 (127.0.103.67) to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 5 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:26.326198 3902 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 5 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250622 13:53:26.326778 4812 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3377fd0584c546c1a938e3a1204b2be3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:26.330920 4183 raft_consensus.cc:1273] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 5 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250622 13:53:26.331387 4812 consensus_queue.cc:1035] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23b47356efa46c792ab125b568ea95a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:27.807689 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:27.810366 3828 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:27.810562 3828 raft_consensus.cc:604] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:53:27.810990 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:27.811203 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:27.994377 4926 raft_consensus.cc:991] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: : Instructing follower d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:27.994546 4926 raft_consensus.cc:1079] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Signalling peer d13e1e991fdc47899aac5b7c33fbbbd9 to start an election
I20250622 13:53:27.995085 3753 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:53780
I20250622 13:53:27.995280 3753 raft_consensus.cc:491] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:27.995386 3753 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:27.997527 3753 raft_consensus.cc:513] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:27.998190 3753 leader_election.cc:290] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 4 election: Requested vote from peers 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66:34391), feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925)
I20250622 13:53:27.998590 4258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
I20250622 13:53:27.998593 3828 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
I20250622 13:53:27.998798 4258 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:27.998836 3828 raft_consensus.cc:3053] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Stepping down as leader of term 3
I20250622 13:53:27.998927 3828 raft_consensus.cc:738] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:27.999155 3828 consensus_queue.cc:260] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:27.999413 3828 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:53:28.000775 4258 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 4.
I20250622 13:53:28.001086 3828 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d13e1e991fdc47899aac5b7c33fbbbd9 in term 4.
I20250622 13:53:28.001238 3725 leader_election.cc:304] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d13e1e991fdc47899aac5b7c33fbbbd9, feaebd1ce3104a2fa3cc207fd3c23468; no voters:
I20250622 13:53:28.001550 4840 raft_consensus.cc:2802] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 13:53:28.001863 4840 raft_consensus.cc:695] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Becoming Leader. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:28.002089 4840 consensus_queue.cc:237] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:28.004091 3651 catalog_manager.cc:5582] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 reported cstate change: term changed from 3 to 4, leader changed from 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66) to d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65). New cstate: current_term: 4 leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:28.539335 4258 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250622 13:53:28.539875 4840 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:28.543792 3828 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Refusing update from remote peer d13e1e991fdc47899aac5b7c33fbbbd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250622 13:53:28.544392 4840 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250622 13:53:29.811600 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:29.814275 3753 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59"
dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
mode: GRACEFUL
new_leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:53844
I20250622 13:53:29.814522 3753 raft_consensus.cc:604] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Received request to transfer leadership to TS 946d416d6f4a4b0ea6a77b93cd6f58f6
I20250622 13:53:29.815063 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:29.815280 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:29.955070 4939 raft_consensus.cc:991] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: : Instructing follower 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:29.955245 4939 raft_consensus.cc:1079] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Signalling peer 946d416d6f4a4b0ea6a77b93cd6f58f6 to start an election
I20250622 13:53:29.955822 3828 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
from {username='slave'} at 127.0.0.1:45606
I20250622 13:53:29.956004 3828 raft_consensus.cc:491] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:29.956089 3828 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:29.958011 3828 raft_consensus.cc:513] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:29.958643 3828 leader_election.cc:290] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 5 election: Requested vote from peers d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517), feaebd1ce3104a2fa3cc207fd3c23468 (127.0.103.69:42925)
I20250622 13:53:29.959147 4258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "feaebd1ce3104a2fa3cc207fd3c23468"
I20250622 13:53:29.959122 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5c429ec7a7a4a7c8b9b50ae4be5ff59" candidate_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
I20250622 13:53:29.959343 4258 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:29.959371 3753 raft_consensus.cc:3053] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Stepping down as leader of term 4
I20250622 13:53:29.959481 3753 raft_consensus.cc:738] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d13e1e991fdc47899aac5b7c33fbbbd9, State: Running, Role: LEADER
I20250622 13:53:29.959748 3753 consensus_queue.cc:260] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:29.960072 3753 raft_consensus.cc:3058] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 4 FOLLOWER]: Advancing to term 5
I20250622 13:53:29.962222 4258 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 5.
I20250622 13:53:29.962779 3753 raft_consensus.cc:2466] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946d416d6f4a4b0ea6a77b93cd6f58f6 in term 5.
I20250622 13:53:29.962745 3800 leader_election.cc:304] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946d416d6f4a4b0ea6a77b93cd6f58f6, feaebd1ce3104a2fa3cc207fd3c23468; no voters:
I20250622 13:53:29.963107 4926 raft_consensus.cc:2802] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Leader election won for term 5
I20250622 13:53:29.963513 4926 raft_consensus.cc:695] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 LEADER]: Becoming Leader. State: Replica: 946d416d6f4a4b0ea6a77b93cd6f58f6, State: Running, Role: LEADER
I20250622 13:53:29.963763 4926 consensus_queue.cc:237] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } }
I20250622 13:53:29.966084 3652 catalog_manager.cc:5582] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 reported cstate change: term changed from 4 to 5, leader changed from d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65) to 946d416d6f4a4b0ea6a77b93cd6f58f6 (127.0.103.66). New cstate: current_term: 5 leader_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:30.368744 3753 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250622 13:53:30.369387 4926 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:30.374238 4259 raft_consensus.cc:1273] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Refusing update from remote peer 946d416d6f4a4b0ea6a77b93cd6f58f6: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250622 13:53:30.374884 4941 consensus_queue.cc:1035] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feaebd1ce3104a2fa3cc207fd3c23468" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250622 13:53:31.815722 3710 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:31.818542 3828 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b045aeb5889a4b88b8ee72370f2a7e39"
dest_uuid: "946d416d6f4a4b0ea6a77b93cd6f58f6"
mode: GRACEFUL
new_leader_uuid: "d13e1e991fdc47899aac5b7c33fbbbd9"
from {username='slave'} at 127.0.0.1:45666
I20250622 13:53:31.818742 3828 raft_consensus.cc:604] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 LEADER]: Received request to transfer leadership to TS d13e1e991fdc47899aac5b7c33fbbbd9
I20250622 13:53:31.819228 3710 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:31.819415 3710 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:195: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250622 13:53:31.976490 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:53:31.993510 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:31.993789 413 tablet_replica.cc:331] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:31.993979 413 raft_consensus.cc:2241] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:31.994154 413 raft_consensus.cc:2270] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:31.994784 413 tablet_replica.cc:331] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:31.994935 413 raft_consensus.cc:2241] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:31.995100 413 raft_consensus.cc:2270] T 1aa8fa3d766949b99d0538f8b76498d5 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:31.995558 413 tablet_replica.cc:331] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:31.995720 413 raft_consensus.cc:2241] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 LEADER]: Raft consensus shutting down.
I20250622 13:53:31.995971 413 raft_consensus.cc:2270] T bac2340a96e94c219f59c12945eb8bf6 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:31.996393 413 tablet_replica.cc:331] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:31.996508 413 raft_consensus.cc:2241] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:53:31.996718 413 raft_consensus.cc:2270] T d317ca8305314d45b47fa1014f3b5162 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:31.997113 413 tablet_replica.cc:331] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:31.997233 413 raft_consensus.cc:2241] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:31.997493 413 raft_consensus.cc:2270] T c3c17332dc1c4acaa727cf9b2fb8ed8e P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:31.997893 413 tablet_replica.cc:331] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9: stopping tablet replica
I20250622 13:53:31.997997 413 raft_consensus.cc:2241] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:31.998109 413 raft_consensus.cc:2270] T b045aeb5889a4b88b8ee72370f2a7e39 P d13e1e991fdc47899aac5b7c33fbbbd9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.014407 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
W20250622 13:53:32.028434 4299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:40517: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20250622 13:53:32.029582 4299 consensus_peers.cc:487] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 -> Peer d13e1e991fdc47899aac5b7c33fbbbd9 (127.0.103.65:40517): Couldn't send request to peer d13e1e991fdc47899aac5b7c33fbbbd9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:40517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:32.035678 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:53:32.045083 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:32.045346 413 tablet_replica.cc:331] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:32.045552 413 raft_consensus.cc:2241] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.045781 413 raft_consensus.cc:2270] T 677dddfcdf864e7f9b997d196b58bc78 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.046370 413 tablet_replica.cc:331] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:32.046525 413 raft_consensus.cc:2241] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.046669 413 raft_consensus.cc:2270] T 1aa8fa3d766949b99d0538f8b76498d5 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.047113 413 tablet_replica.cc:331] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:32.047235 413 raft_consensus.cc:2241] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.047536 413 raft_consensus.cc:2270] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.047940 413 tablet_replica.cc:331] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:32.048069 413 raft_consensus.cc:2241] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.048306 413 raft_consensus.cc:2270] T 6fd3fa41ece64410b7448f59d5db9ba7 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.048703 413 tablet_replica.cc:331] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:32.048794 413 raft_consensus.cc:2241] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.048972 413 raft_consensus.cc:2270] T b045aeb5889a4b88b8ee72370f2a7e39 P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.049337 413 tablet_replica.cc:331] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6: stopping tablet replica
I20250622 13:53:32.049468 413 raft_consensus.cc:2241] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.049793 413 raft_consensus.cc:2270] T f10daa2b314a4660bd36fa7902789a8e P 946d416d6f4a4b0ea6a77b93cd6f58f6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.065806 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:53:32.071291 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:53:32.096328 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:32.096601 413 tablet_replica.cc:331] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:32.096786 413 raft_consensus.cc:2241] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.097057 413 raft_consensus.cc:2270] T bac2340a96e94c219f59c12945eb8bf6 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.097716 413 tablet_replica.cc:331] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:32.097856 413 raft_consensus.cc:2241] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.098058 413 raft_consensus.cc:2270] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 3377fd0584c546c1a938e3a1204b2be3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.098551 413 tablet_replica.cc:331] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:32.098680 413 raft_consensus.cc:2241] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.098803 413 raft_consensus.cc:2270] T 6fd3fa41ece64410b7448f59d5db9ba7 P 3377fd0584c546c1a938e3a1204b2be3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.099234 413 tablet_replica.cc:331] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:32.099360 413 raft_consensus.cc:2241] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.099476 413 raft_consensus.cc:2270] T d317ca8305314d45b47fa1014f3b5162 P 3377fd0584c546c1a938e3a1204b2be3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.099884 413 tablet_replica.cc:331] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:32.100006 413 raft_consensus.cc:2241] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.100268 413 raft_consensus.cc:2270] T c9260a075f104769bc81ca7eb0c76cb2 P 3377fd0584c546c1a938e3a1204b2be3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.100698 413 tablet_replica.cc:331] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3: stopping tablet replica
I20250622 13:53:32.100806 413 raft_consensus.cc:2241] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 5 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.101023 413 raft_consensus.cc:2270] T c3c17332dc1c4acaa727cf9b2fb8ed8e P 3377fd0584c546c1a938e3a1204b2be3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.118782 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:53:32.123629 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250622 13:53:32.132109 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:32.132351 413 tablet_replica.cc:331] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a: stopping tablet replica
I20250622 13:53:32.132544 413 raft_consensus.cc:2241] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.132694 413 raft_consensus.cc:2270] T 6fd3fa41ece64410b7448f59d5db9ba7 P d23b47356efa46c792ab125b568ea95a [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.133239 413 tablet_replica.cc:331] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a: stopping tablet replica
I20250622 13:53:32.133397 413 raft_consensus.cc:2241] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.133574 413 raft_consensus.cc:2270] T 91736fef6df543c4953497186dfe1c5e P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.133976 413 tablet_replica.cc:331] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a: stopping tablet replica
I20250622 13:53:32.134074 413 raft_consensus.cc:2241] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 3 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.134303 413 raft_consensus.cc:2270] T 5246e7fae8d44ba6a5e3ac826f9988f4 P d23b47356efa46c792ab125b568ea95a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.134680 413 tablet_replica.cc:331] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a: stopping tablet replica
I20250622 13:53:32.134770 413 raft_consensus.cc:2241] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.134860 413 raft_consensus.cc:2270] T b045aeb5889a4b88b8ee72370f2a7e39 P d23b47356efa46c792ab125b568ea95a [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.135217 413 tablet_replica.cc:331] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a: stopping tablet replica
I20250622 13:53:32.135304 413 raft_consensus.cc:2241] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.135460 413 raft_consensus.cc:2270] T c9260a075f104769bc81ca7eb0c76cb2 P d23b47356efa46c792ab125b568ea95a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.135814 413 tablet_replica.cc:331] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a: stopping tablet replica
I20250622 13:53:32.135949 413 raft_consensus.cc:2241] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.136209 413 raft_consensus.cc:2270] T f10daa2b314a4660bd36fa7902789a8e P d23b47356efa46c792ab125b568ea95a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.150926 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250622 13:53:32.154824 413 tablet_server.cc:178] TabletServer@127.0.103.69:0 shutting down...
I20250622 13:53:32.164069 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:32.164327 413 tablet_replica.cc:331] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468: stopping tablet replica
I20250622 13:53:32.164517 413 raft_consensus.cc:2241] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.164721 413 raft_consensus.cc:2270] T c3c17332dc1c4acaa727cf9b2fb8ed8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.165270 413 tablet_replica.cc:331] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468: stopping tablet replica
I20250622 13:53:32.165422 413 raft_consensus.cc:2241] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.165607 413 raft_consensus.cc:2270] T f10daa2b314a4660bd36fa7902789a8e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.166043 413 tablet_replica.cc:331] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468: stopping tablet replica
I20250622 13:53:32.166172 413 raft_consensus.cc:2241] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.166299 413 raft_consensus.cc:2270] T d317ca8305314d45b47fa1014f3b5162 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.166711 413 tablet_replica.cc:331] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468: stopping tablet replica
I20250622 13:53:32.166832 413 raft_consensus.cc:2241] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.166951 413 raft_consensus.cc:2270] T b5c429ec7a7a4a7c8b9b50ae4be5ff59 P feaebd1ce3104a2fa3cc207fd3c23468 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.167332 413 tablet_replica.cc:331] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468: stopping tablet replica
I20250622 13:53:32.167452 413 raft_consensus.cc:2241] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.167701 413 raft_consensus.cc:2270] T 91736fef6df543c4953497186dfe1c5e P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.168112 413 tablet_replica.cc:331] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468: stopping tablet replica
I20250622 13:53:32.168224 413 raft_consensus.cc:2241] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.168437 413 raft_consensus.cc:2270] T 677dddfcdf864e7f9b997d196b58bc78 P feaebd1ce3104a2fa3cc207fd3c23468 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.199931 413 tablet_server.cc:195] TabletServer@127.0.103.69:0 shutdown complete.
I20250622 13:53:32.203879 413 tablet_server.cc:178] TabletServer@127.0.103.70:0 shutting down...
I20250622 13:53:32.212890 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:32.213127 413 tablet_replica.cc:331] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79: stopping tablet replica
I20250622 13:53:32.213317 413 raft_consensus.cc:2241] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.213587 413 raft_consensus.cc:2270] T 5246e7fae8d44ba6a5e3ac826f9988f4 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.214164 413 tablet_replica.cc:331] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79: stopping tablet replica
I20250622 13:53:32.214310 413 raft_consensus.cc:2241] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.214521 413 raft_consensus.cc:2270] T c9260a075f104769bc81ca7eb0c76cb2 P 52c21167280a4d669123bb1ed7675d79 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.214960 413 tablet_replica.cc:331] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79: stopping tablet replica
I20250622 13:53:32.215090 413 raft_consensus.cc:2241] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.215260 413 raft_consensus.cc:2270] T bac2340a96e94c219f59c12945eb8bf6 P 52c21167280a4d669123bb1ed7675d79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.215672 413 tablet_replica.cc:331] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79: stopping tablet replica
I20250622 13:53:32.215793 413 raft_consensus.cc:2241] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.215960 413 raft_consensus.cc:2270] T 677dddfcdf864e7f9b997d196b58bc78 P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.216361 413 tablet_replica.cc:331] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79: stopping tablet replica
I20250622 13:53:32.216487 413 raft_consensus.cc:2241] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.216729 413 raft_consensus.cc:2270] T 1aa8fa3d766949b99d0538f8b76498d5 P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.217113 413 tablet_replica.cc:331] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79: stopping tablet replica
I20250622 13:53:32.217231 413 raft_consensus.cc:2241] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:32.217569 413 raft_consensus.cc:2270] T 91736fef6df543c4953497186dfe1c5e P 52c21167280a4d669123bb1ed7675d79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.232203 413 tablet_server.cc:195] TabletServer@127.0.103.70:0 shutdown complete.
I20250622 13:53:32.236142 413 master.cc:561] Master@127.0.103.126:37277 shutting down...
I20250622 13:53:32.244321 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:53:32.244556 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:32.244684 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P ff80a419fea7490e880cbf4be482cb37: stopping tablet replica
I20250622 13:53:32.258049 413 master.cc:583] Master@127.0.103.126:37277 shutdown complete.
I20250622 13:53:32.265166 413 test_util.cc:183] -----------------------------------------------
I20250622 13:53:32.265239 413 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750600281110842-413-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (37266 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20250622 13:53:32.267102 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:32833
I20250622 13:53:32.267608 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:32.269852 4961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:32.269876 4962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:32.270614 4964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:32.274713 413 server_base.cc:1048] running on GCE node
I20250622 13:53:32.274940 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:32.274992 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:32.275035 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600412275030 us; error 0 us; skew 500 ppm
I20250622 13:53:32.275183 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:32.281277 413 webserver.cc:469] Webserver started at http://127.0.103.126:33511/ using document root <none> and password file <none>
I20250622 13:53:32.281550 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:32.281622 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:32.281750 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:32.282385 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "3bda0cf81e724234a2793350f7a3f495"
format_stamp: "Formatted at 2025-06-22 13:53:32 on dist-test-slave-tmv0"
I20250622 13:53:32.284889 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250622 13:53:32.286543 4969 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.286954 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:53:32.287060 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "3bda0cf81e724234a2793350f7a3f495"
format_stamp: "Formatted at 2025-06-22 13:53:32 on dist-test-slave-tmv0"
I20250622 13:53:32.287182 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:32.300809 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:32.301337 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:32.312283 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:32833
I20250622 13:53:32.312296 5020 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:32833 every 8 connection(s)
I20250622 13:53:32.313532 5021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:32.317272 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495: Bootstrap starting.
I20250622 13:53:32.318568 5021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:32.320087 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495: No bootstrap required, opened a new log
I20250622 13:53:32.320658 5021 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bda0cf81e724234a2793350f7a3f495" member_type: VOTER }
I20250622 13:53:32.320775 5021 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:32.320827 5021 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bda0cf81e724234a2793350f7a3f495, State: Initialized, Role: FOLLOWER
I20250622 13:53:32.321007 5021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bda0cf81e724234a2793350f7a3f495" member_type: VOTER }
I20250622 13:53:32.321105 5021 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:53:32.321161 5021 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:53:32.321233 5021 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:32.322918 5021 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bda0cf81e724234a2793350f7a3f495" member_type: VOTER }
I20250622 13:53:32.323086 5021 leader_election.cc:304] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bda0cf81e724234a2793350f7a3f495; no voters:
I20250622 13:53:32.323453 5021 leader_election.cc:290] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:53:32.323608 5024 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:32.324035 5024 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 1 LEADER]: Becoming Leader. State: Replica: 3bda0cf81e724234a2793350f7a3f495, State: Running, Role: LEADER
I20250622 13:53:32.324290 5021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:53:32.324262 5024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bda0cf81e724234a2793350f7a3f495" member_type: VOTER }
I20250622 13:53:32.325047 5026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bda0cf81e724234a2793350f7a3f495. Latest consensus state: current_term: 1 leader_uuid: "3bda0cf81e724234a2793350f7a3f495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bda0cf81e724234a2793350f7a3f495" member_type: VOTER } }
I20250622 13:53:32.325176 5026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [sys.catalog]: This master's current role is: LEADER
I20250622 13:53:32.325390 5025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bda0cf81e724234a2793350f7a3f495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bda0cf81e724234a2793350f7a3f495" member_type: VOTER } }
I20250622 13:53:32.325625 5025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [sys.catalog]: This master's current role is: LEADER
I20250622 13:53:32.329401 5029 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:53:32.331143 5029 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:53:32.331512 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:53:32.334053 5029 catalog_manager.cc:1349] Generated new cluster ID: 3824a1798216496594cb06d1b38b967f
I20250622 13:53:32.334136 5029 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:53:32.346407 5029 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:53:32.347131 5029 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:53:32.377192 5029 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495: Generated new TSK 0
I20250622 13:53:32.377462 5029 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:53:32.396801 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:32.399258 5042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:32.399503 5043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:32.399917 5045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:32.400404 413 server_base.cc:1048] running on GCE node
I20250622 13:53:32.403144 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:32.403209 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:32.403232 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600412403230 us; error 0 us; skew 500 ppm
I20250622 13:53:32.403348 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:32.404572 413 webserver.cc:469] Webserver started at http://127.0.103.65:39077/ using document root <none> and password file <none>
I20250622 13:53:32.404762 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:32.404814 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:32.404907 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:32.405545 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "041d5c7800494072b40a987b103573d7"
format_stamp: "Formatted at 2025-06-22 13:53:32 on dist-test-slave-tmv0"
I20250622 13:53:32.407891 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250622 13:53:32.409392 5050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.409742 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:53:32.411020 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "041d5c7800494072b40a987b103573d7"
format_stamp: "Formatted at 2025-06-22 13:53:32 on dist-test-slave-tmv0"
I20250622 13:53:32.411139 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:32.418835 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:32.419329 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:32.419873 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:32.420686 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:32.420750 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.420846 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:32.420917 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.433061 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45501
I20250622 13:53:32.433092 5112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:45501 every 8 connection(s)
I20250622 13:53:32.434825 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:32.437539 5117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:32.437600 5118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:32.440140 413 server_base.cc:1048] running on GCE node
W20250622 13:53:32.440999 5120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:32.441334 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:32.441452 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:32.441514 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600412441491 us; error 0 us; skew 500 ppm
I20250622 13:53:32.441712 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:32.441731 5113 heartbeater.cc:344] Connected to a master server at 127.0.103.126:32833
I20250622 13:53:32.441879 5113 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:32.442181 5113 heartbeater.cc:507] Master 127.0.103.126:32833 requested a full tablet report, sending...
I20250622 13:53:32.443044 4986 ts_manager.cc:194] Registered new tserver with Master: 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501)
I20250622 13:53:32.443184 413 webserver.cc:469] Webserver started at http://127.0.103.66:36353/ using document root <none> and password file <none>
I20250622 13:53:32.443419 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:32.443516 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:32.443670 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:32.444406 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "1af80fe6712e4bd0b34e93169f5a0705"
format_stamp: "Formatted at 2025-06-22 13:53:32 on dist-test-slave-tmv0"
I20250622 13:53:32.444494 4986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40408
I20250622 13:53:32.447075 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20250622 13:53:32.448473 5125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.448814 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:53:32.448920 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "1af80fe6712e4bd0b34e93169f5a0705"
format_stamp: "Formatted at 2025-06-22 13:53:32 on dist-test-slave-tmv0"
I20250622 13:53:32.449033 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:32.472142 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:32.472662 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:32.473214 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:32.474030 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:32.474097 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.474190 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:32.474244 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.486501 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:46717
I20250622 13:53:32.486537 5187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:46717 every 8 connection(s)
I20250622 13:53:32.488122 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:32.491034 5191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:32.491132 5192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:32.492136 5194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:32.492695 413 server_base.cc:1048] running on GCE node
I20250622 13:53:32.492965 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:32.493068 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:32.493134 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600412493116 us; error 0 us; skew 500 ppm
I20250622 13:53:32.493321 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:32.494540 5188 heartbeater.cc:344] Connected to a master server at 127.0.103.126:32833
I20250622 13:53:32.494704 5188 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:32.494800 413 webserver.cc:469] Webserver started at http://127.0.103.67:37911/ using document root <none> and password file <none>
I20250622 13:53:32.495038 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:32.495055 5188 heartbeater.cc:507] Master 127.0.103.126:32833 requested a full tablet report, sending...
I20250622 13:53:32.495119 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:32.495262 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:32.495926 4986 ts_manager.cc:194] Registered new tserver with Master: 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717)
I20250622 13:53:32.495967 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "4734b8dc14974411ad459aaf6a399c2f"
format_stamp: "Formatted at 2025-06-22 13:53:32 on dist-test-slave-tmv0"
I20250622 13:53:32.497026 4986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40422
I20250622 13:53:32.498723 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:32.500180 5199 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.500566 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:53:32.500665 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "4734b8dc14974411ad459aaf6a399c2f"
format_stamp: "Formatted at 2025-06-22 13:53:32 on dist-test-slave-tmv0"
I20250622 13:53:32.500785 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:32.519994 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:32.520565 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:32.521142 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:32.521930 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:32.522001 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.522104 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:32.522152 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:32.535295 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:38299
I20250622 13:53:32.535328 5261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:38299 every 8 connection(s)
I20250622 13:53:32.541951 5262 heartbeater.cc:344] Connected to a master server at 127.0.103.126:32833
I20250622 13:53:32.542126 5262 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:32.542497 5262 heartbeater.cc:507] Master 127.0.103.126:32833 requested a full tablet report, sending...
I20250622 13:53:32.543339 4986 ts_manager.cc:194] Registered new tserver with Master: 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:32.544044 4986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40428
I20250622 13:53:32.544270 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008038344s
I20250622 13:53:33.445770 5113 heartbeater.cc:499] Master 127.0.103.126:32833 was elected leader, sending a full tablet report...
I20250622 13:53:33.498265 5188 heartbeater.cc:499] Master 127.0.103.126:32833 was elected leader, sending a full tablet report...
I20250622 13:53:33.545286 5262 heartbeater.cc:499] Master 127.0.103.126:32833 was elected leader, sending a full tablet report...
I20250622 13:53:33.572009 413 test_util.cc:276] Using random seed: 1808493440
I20250622 13:53:33.583222 4986 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40438:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:53:33.583827 4986 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:53:33.605046 5227 tablet_service.cc:1468] Processing CreateTablet for tablet c4e25c6178ed498499bc9afc69eab8d5 (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250622 13:53:33.605521 5227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e25c6178ed498499bc9afc69eab8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.605773 5225 tablet_service.cc:1468] Processing CreateTablet for tablet cfe0044d7bce4e93813f9883aece8502 (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250622 13:53:33.606339 5225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfe0044d7bce4e93813f9883aece8502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.607939 5078 tablet_service.cc:1468] Processing CreateTablet for tablet c4e25c6178ed498499bc9afc69eab8d5 (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250622 13:53:33.608377 5078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e25c6178ed498499bc9afc69eab8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.605224 5226 tablet_service.cc:1468] Processing CreateTablet for tablet 50e27305bc8f4295a4a37be272baae9e (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250622 13:53:33.612498 5226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e27305bc8f4295a4a37be272baae9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.612803 5077 tablet_service.cc:1468] Processing CreateTablet for tablet 50e27305bc8f4295a4a37be272baae9e (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250622 13:53:33.613106 5077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e27305bc8f4295a4a37be272baae9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.617077 5283 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: Bootstrap starting.
I20250622 13:53:33.617049 5076 tablet_service.cc:1468] Processing CreateTablet for tablet cfe0044d7bce4e93813f9883aece8502 (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250622 13:53:33.617486 5076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfe0044d7bce4e93813f9883aece8502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.618826 5283 tablet_bootstrap.cc:654] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.625169 5282 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f: Bootstrap starting.
I20250622 13:53:33.627372 5282 tablet_bootstrap.cc:654] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.627744 5283 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: No bootstrap required, opened a new log
I20250622 13:53:33.627962 5283 ts_tablet_manager.cc:1397] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20250622 13:53:33.628813 5283 raft_consensus.cc:357] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.629014 5283 raft_consensus.cc:383] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.629118 5283 raft_consensus.cc:738] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041d5c7800494072b40a987b103573d7, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.629494 5282 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f: No bootstrap required, opened a new log
I20250622 13:53:33.629386 5283 consensus_queue.cc:260] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.629715 5282 ts_tablet_manager.cc:1397] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250622 13:53:33.630138 5153 tablet_service.cc:1468] Processing CreateTablet for tablet c4e25c6178ed498499bc9afc69eab8d5 (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250622 13:53:33.630519 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e25c6178ed498499bc9afc69eab8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.630749 5282 raft_consensus.cc:357] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.631000 5282 raft_consensus.cc:383] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.631143 5282 raft_consensus.cc:738] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4734b8dc14974411ad459aaf6a399c2f, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.631490 5282 consensus_queue.cc:260] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.632427 5282 ts_tablet_manager.cc:1428] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:53:33.632822 5282 tablet_bootstrap.cc:492] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f: Bootstrap starting.
I20250622 13:53:33.633765 5152 tablet_service.cc:1468] Processing CreateTablet for tablet 50e27305bc8f4295a4a37be272baae9e (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250622 13:53:33.634186 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e27305bc8f4295a4a37be272baae9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.635201 5282 tablet_bootstrap.cc:654] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.637689 5151 tablet_service.cc:1468] Processing CreateTablet for tablet cfe0044d7bce4e93813f9883aece8502 (DEFAULT_TABLE table=test-workload [id=a990b1eb84604eaeaff0578e61433b84]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250622 13:53:33.637981 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfe0044d7bce4e93813f9883aece8502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:33.653012 5289 tablet_bootstrap.cc:492] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705: Bootstrap starting.
I20250622 13:53:33.653321 5283 ts_tablet_manager.cc:1428] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: Time spent starting tablet: real 0.025s user 0.008s sys 0.016s
I20250622 13:53:33.654239 5283 tablet_bootstrap.cc:492] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7: Bootstrap starting.
I20250622 13:53:33.656086 5289 tablet_bootstrap.cc:654] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.656597 5283 tablet_bootstrap.cc:654] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.658281 5282 tablet_bootstrap.cc:492] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f: No bootstrap required, opened a new log
I20250622 13:53:33.658349 5289 tablet_bootstrap.cc:492] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705: No bootstrap required, opened a new log
I20250622 13:53:33.658437 5282 ts_tablet_manager.cc:1397] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.012s
I20250622 13:53:33.658507 5289 ts_tablet_manager.cc:1397] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20250622 13:53:33.659016 5283 tablet_bootstrap.cc:492] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7: No bootstrap required, opened a new log
I20250622 13:53:33.659155 5283 ts_tablet_manager.cc:1397] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250622 13:53:33.659269 5282 raft_consensus.cc:357] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.659353 5289 raft_consensus.cc:357] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.659480 5282 raft_consensus.cc:383] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.659518 5289 raft_consensus.cc:383] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.659572 5282 raft_consensus.cc:738] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4734b8dc14974411ad459aaf6a399c2f, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.659596 5289 raft_consensus.cc:738] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af80fe6712e4bd0b34e93169f5a0705, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.659804 5289 consensus_queue.cc:260] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.659803 5282 consensus_queue.cc:260] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.659904 5283 raft_consensus.cc:357] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.660176 5283 raft_consensus.cc:383] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.660298 5283 raft_consensus.cc:738] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041d5c7800494072b40a987b103573d7, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.660421 5282 ts_tablet_manager.cc:1428] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:53:33.660533 5283 consensus_queue.cc:260] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.660789 5282 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f: Bootstrap starting.
I20250622 13:53:33.661590 5283 ts_tablet_manager.cc:1428] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:53:33.661881 5283 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: Bootstrap starting.
I20250622 13:53:33.662994 5282 tablet_bootstrap.cc:654] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.663556 5283 tablet_bootstrap.cc:654] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.668995 5289 ts_tablet_manager.cc:1428] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20250622 13:53:33.669795 5289 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: Bootstrap starting.
I20250622 13:53:33.670166 5282 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f: No bootstrap required, opened a new log
I20250622 13:53:33.670351 5282 ts_tablet_manager.cc:1397] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20250622 13:53:33.670523 5283 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: No bootstrap required, opened a new log
I20250622 13:53:33.670771 5283 ts_tablet_manager.cc:1397] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.007s
I20250622 13:53:33.671204 5282 raft_consensus.cc:357] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.671445 5282 raft_consensus.cc:383] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.671440 5283 raft_consensus.cc:357] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.671588 5282 raft_consensus.cc:738] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4734b8dc14974411ad459aaf6a399c2f, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.671629 5289 tablet_bootstrap.cc:654] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.671650 5283 raft_consensus.cc:383] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.671808 5283 raft_consensus.cc:738] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041d5c7800494072b40a987b103573d7, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.671898 5282 consensus_queue.cc:260] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.672403 5282 ts_tablet_manager.cc:1428] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:53:33.672039 5283 consensus_queue.cc:260] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.674243 5283 ts_tablet_manager.cc:1428] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250622 13:53:33.675433 5289 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: No bootstrap required, opened a new log
I20250622 13:53:33.675592 5289 ts_tablet_manager.cc:1397] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20250622 13:53:33.676143 5289 raft_consensus.cc:357] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.676337 5289 raft_consensus.cc:383] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.676424 5289 raft_consensus.cc:738] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af80fe6712e4bd0b34e93169f5a0705, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.676640 5289 consensus_queue.cc:260] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.676975 5289 ts_tablet_manager.cc:1428] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250622 13:53:33.677253 5289 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705: Bootstrap starting.
I20250622 13:53:33.678638 5289 tablet_bootstrap.cc:654] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:33.680192 5289 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705: No bootstrap required, opened a new log
I20250622 13:53:33.680313 5289 ts_tablet_manager.cc:1397] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:53:33.680840 5289 raft_consensus.cc:357] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.680974 5289 raft_consensus.cc:383] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:33.681056 5289 raft_consensus.cc:738] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af80fe6712e4bd0b34e93169f5a0705, State: Initialized, Role: FOLLOWER
I20250622 13:53:33.681280 5289 consensus_queue.cc:260] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.681763 5289 ts_tablet_manager.cc:1428] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20250622 13:53:33.685115 5114 tablet.cc:2378] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:53:33.695643 5286 raft_consensus.cc:491] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:33.695835 5286 raft_consensus.cc:513] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.696696 5286 leader_election.cc:290] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:33.703507 5292 raft_consensus.cc:491] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:33.703677 5292 raft_consensus.cc:513] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.704243 5237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4734b8dc14974411ad459aaf6a399c2f" is_pre_election: true
I20250622 13:53:33.704283 5163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705" is_pre_election: true
I20250622 13:53:33.704476 5237 raft_consensus.cc:2466] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 0.
I20250622 13:53:33.704668 5292 leader_election.cc:290] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:33.704486 5163 raft_consensus.cc:2466] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 0.
I20250622 13:53:33.705485 5054 leader_election.cc:304] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d5c7800494072b40a987b103573d7, 4734b8dc14974411ad459aaf6a399c2f; no voters:
I20250622 13:53:33.705833 5286 raft_consensus.cc:2802] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:33.705938 5286 raft_consensus.cc:491] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:33.706029 5286 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.708572 5286 raft_consensus.cc:513] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.709676 5237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4734b8dc14974411ad459aaf6a399c2f"
I20250622 13:53:33.709672 5163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
I20250622 13:53:33.709887 5237 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.709903 5163 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.712457 5286 leader_election.cc:290] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:33.712513 5088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041d5c7800494072b40a987b103573d7" is_pre_election: true
I20250622 13:53:33.712675 5286 raft_consensus.cc:491] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:33.712787 5088 raft_consensus.cc:2391] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1af80fe6712e4bd0b34e93169f5a0705 in current term 1: Already voted for candidate 041d5c7800494072b40a987b103573d7 in this term.
I20250622 13:53:33.712833 5286 raft_consensus.cc:513] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.712795 5237 raft_consensus.cc:2466] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 1.
I20250622 13:53:33.713207 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4734b8dc14974411ad459aaf6a399c2f" is_pre_election: true
I20250622 13:53:33.713399 5054 leader_election.cc:304] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d5c7800494072b40a987b103573d7, 4734b8dc14974411ad459aaf6a399c2f; no voters:
I20250622 13:53:33.713497 5236 raft_consensus.cc:2391] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1af80fe6712e4bd0b34e93169f5a0705 in current term 1: Already voted for candidate 041d5c7800494072b40a987b103573d7 in this term.
I20250622 13:53:33.713500 5286 leader_election.cc:290] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:33.713961 5286 raft_consensus.cc:2802] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:33.714051 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfe0044d7bce4e93813f9883aece8502" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4734b8dc14974411ad459aaf6a399c2f" is_pre_election: true
I20250622 13:53:33.713970 5129 leader_election.cc:304] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1af80fe6712e4bd0b34e93169f5a0705; no voters: 041d5c7800494072b40a987b103573d7, 4734b8dc14974411ad459aaf6a399c2f
I20250622 13:53:33.714232 5286 raft_consensus.cc:695] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Becoming Leader. State: Replica: 041d5c7800494072b40a987b103573d7, State: Running, Role: LEADER
I20250622 13:53:33.714290 5236 raft_consensus.cc:2466] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 0.
I20250622 13:53:33.713938 5163 raft_consensus.cc:2466] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 1.
I20250622 13:53:33.714490 5286 consensus_queue.cc:237] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.714609 5292 raft_consensus.cc:2747] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250622 13:53:33.714951 5054 leader_election.cc:304] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d5c7800494072b40a987b103573d7, 4734b8dc14974411ad459aaf6a399c2f; no voters:
I20250622 13:53:33.715816 5301 raft_consensus.cc:2802] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:33.715963 5301 raft_consensus.cc:491] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:33.716075 5301 raft_consensus.cc:3058] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.716328 5163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfe0044d7bce4e93813f9883aece8502" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705" is_pre_election: true
I20250622 13:53:33.716523 5163 raft_consensus.cc:2466] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 0.
I20250622 13:53:33.716838 5287 raft_consensus.cc:491] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:33.716986 5287 raft_consensus.cc:513] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.717821 5287 leader_election.cc:290] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501), 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717)
I20250622 13:53:33.718401 4985 catalog_manager.cc:5582] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 041d5c7800494072b40a987b103573d7 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "041d5c7800494072b40a987b103573d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:33.719990 5301 raft_consensus.cc:513] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.720782 5163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfe0044d7bce4e93813f9883aece8502" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
I20250622 13:53:33.720933 5163 raft_consensus.cc:3058] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.723798 5163 raft_consensus.cc:2466] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 1.
I20250622 13:53:33.724205 5053 leader_election.cc:304] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d5c7800494072b40a987b103573d7, 1af80fe6712e4bd0b34e93169f5a0705; no voters:
I20250622 13:53:33.724364 5237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfe0044d7bce4e93813f9883aece8502" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4734b8dc14974411ad459aaf6a399c2f"
I20250622 13:53:33.724550 5237 raft_consensus.cc:3058] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.724787 5286 raft_consensus.cc:2802] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:33.724886 5301 leader_election.cc:290] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:33.724962 5286 raft_consensus.cc:695] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Becoming Leader. State: Replica: 041d5c7800494072b40a987b103573d7, State: Running, Role: LEADER
I20250622 13:53:33.724995 5088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e25c6178ed498499bc9afc69eab8d5" candidate_uuid: "4734b8dc14974411ad459aaf6a399c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041d5c7800494072b40a987b103573d7" is_pre_election: true
I20250622 13:53:33.725142 5088 raft_consensus.cc:2466] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4734b8dc14974411ad459aaf6a399c2f in term 0.
I20250622 13:53:33.725232 5286 consensus_queue.cc:237] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:33.725579 5202 leader_election.cc:304] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d5c7800494072b40a987b103573d7, 4734b8dc14974411ad459aaf6a399c2f; no voters:
I20250622 13:53:33.725812 5287 raft_consensus.cc:2802] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:33.725924 5287 raft_consensus.cc:491] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:33.726020 5287 raft_consensus.cc:3058] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.727311 5237 raft_consensus.cc:2466] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 1.
I20250622 13:53:33.727981 5292 raft_consensus.cc:491] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:33.728123 5292 raft_consensus.cc:513] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.728726 5292 leader_election.cc:290] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:33.729593 5088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e25c6178ed498499bc9afc69eab8d5" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041d5c7800494072b40a987b103573d7" is_pre_election: true
I20250622 13:53:33.729566 5287 raft_consensus.cc:513] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.729852 5088 raft_consensus.cc:2466] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1af80fe6712e4bd0b34e93169f5a0705 in term 0.
I20250622 13:53:33.730094 5163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e25c6178ed498499bc9afc69eab8d5" candidate_uuid: "4734b8dc14974411ad459aaf6a399c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705" is_pre_election: true
I20250622 13:53:33.730290 5163 raft_consensus.cc:2466] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4734b8dc14974411ad459aaf6a399c2f in term 0.
I20250622 13:53:33.730405 5128 leader_election.cc:304] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d5c7800494072b40a987b103573d7, 1af80fe6712e4bd0b34e93169f5a0705; no voters:
I20250622 13:53:33.730671 5292 raft_consensus.cc:2802] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:33.730779 5292 raft_consensus.cc:491] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:33.730784 5287 leader_election.cc:290] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [CANDIDATE]: Term 1 election: Requested vote from peers 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501), 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717)
I20250622 13:53:33.730888 5292 raft_consensus.cc:3058] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.730094 5237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e25c6178ed498499bc9afc69eab8d5" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4734b8dc14974411ad459aaf6a399c2f" is_pre_election: true
I20250622 13:53:33.731235 5163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e25c6178ed498499bc9afc69eab8d5" candidate_uuid: "4734b8dc14974411ad459aaf6a399c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
I20250622 13:53:33.731407 5237 raft_consensus.cc:2391] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1af80fe6712e4bd0b34e93169f5a0705 in current term 1: Already voted for candidate 4734b8dc14974411ad459aaf6a399c2f in this term.
I20250622 13:53:33.731794 5088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e25c6178ed498499bc9afc69eab8d5" candidate_uuid: "4734b8dc14974411ad459aaf6a399c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041d5c7800494072b40a987b103573d7"
I20250622 13:53:33.731950 5088 raft_consensus.cc:3058] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:33.733572 5292 raft_consensus.cc:513] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.734458 5087 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e25c6178ed498499bc9afc69eab8d5" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041d5c7800494072b40a987b103573d7"
I20250622 13:53:33.734650 5088 raft_consensus.cc:2466] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4734b8dc14974411ad459aaf6a399c2f in term 1.
I20250622 13:53:33.734930 5163 raft_consensus.cc:2391] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4734b8dc14974411ad459aaf6a399c2f in current term 1: Already voted for candidate 1af80fe6712e4bd0b34e93169f5a0705 in this term.
I20250622 13:53:33.735157 5202 leader_election.cc:304] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d5c7800494072b40a987b103573d7, 4734b8dc14974411ad459aaf6a399c2f; no voters:
I20250622 13:53:33.735515 5292 leader_election.cc:290] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 1 election: Requested vote from peers 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:33.735432 4985 catalog_manager.cc:5582] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 041d5c7800494072b40a987b103573d7 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "041d5c7800494072b40a987b103573d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:33.735666 5237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e25c6178ed498499bc9afc69eab8d5" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4734b8dc14974411ad459aaf6a399c2f"
I20250622 13:53:33.735783 5287 raft_consensus.cc:2802] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:33.736213 5287 raft_consensus.cc:695] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 LEADER]: Becoming Leader. State: Replica: 4734b8dc14974411ad459aaf6a399c2f, State: Running, Role: LEADER
I20250622 13:53:33.736490 5287 consensus_queue.cc:237] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } }
I20250622 13:53:33.737942 5129 leader_election.cc:304] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1af80fe6712e4bd0b34e93169f5a0705; no voters: 041d5c7800494072b40a987b103573d7, 4734b8dc14974411ad459aaf6a399c2f
I20250622 13:53:33.738211 5292 raft_consensus.cc:2747] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250622 13:53:33.738864 4985 catalog_manager.cc:5582] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67). New cstate: current_term: 1 leader_uuid: "4734b8dc14974411ad459aaf6a399c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:33.742086 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:33.744534 5309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:33.744845 5310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:33.745687 413 server_base.cc:1048] running on GCE node
W20250622 13:53:33.745765 5312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:33.746116 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:33.746187 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:33.746249 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600413746230 us; error 0 us; skew 500 ppm
I20250622 13:53:33.746409 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:33.747743 413 webserver.cc:469] Webserver started at http://127.0.103.68:35439/ using document root <none> and password file <none>
I20250622 13:53:33.747975 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:33.748057 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:33.748205 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:33.748858 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-3-root/instance:
uuid: "2790345be1c04eddbe56c478527ade62"
format_stamp: "Formatted at 2025-06-22 13:53:33 on dist-test-slave-tmv0"
I20250622 13:53:33.751448 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250622 13:53:33.752992 5317 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:33.753345 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:53:33.753494 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-3-root
uuid: "2790345be1c04eddbe56c478527ade62"
format_stamp: "Formatted at 2025-06-22 13:53:33 on dist-test-slave-tmv0"
I20250622 13:53:33.753625 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:33.764372 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:33.764950 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:33.765547 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:33.766520 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:33.766603 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:33.766721 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:33.766790 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:33.779141 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:43953
I20250622 13:53:33.779170 5379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:43953 every 8 connection(s)
I20250622 13:53:33.785980 5380 heartbeater.cc:344] Connected to a master server at 127.0.103.126:32833
I20250622 13:53:33.786129 5380 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:33.786449 5380 heartbeater.cc:507] Master 127.0.103.126:32833 requested a full tablet report, sending...
I20250622 13:53:33.787211 4986 ts_manager.cc:194] Registered new tserver with Master: 2790345be1c04eddbe56c478527ade62 (127.0.103.68:43953)
I20250622 13:53:33.787961 4986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40442
I20250622 13:53:33.799067 5286 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:53:33.800479 5302 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:53:33.802183 5301 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250622 13:53:33.802275 5301 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250622 13:53:33.803241 5286 consensus_queue.cc:1035] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:53:33.813391 5287 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:53:33.817140 5286 consensus_queue.cc:1035] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:53:33.833961 5287 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:53:34.330144 5039 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:34.337960 5087 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e"
dest_uuid: "041d5c7800494072b40a987b103573d7"
mode: GRACEFUL
new_leader_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
from {username='slave'} at 127.0.0.1:60762
I20250622 13:53:34.338706 5088 consensus_queue.cc:237] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.339767 5087 raft_consensus.cc:604] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Received request to transfer leadership to TS 1af80fe6712e4bd0b34e93169f5a0705
I20250622 13:53:34.340643 5162 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:53:34.341300 5236 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:53:34.341220 5392 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:53:34.341229 5039 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:34.342067 5039 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:34.341972 5302 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:53:34.344759 5162 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2790345be1c04eddbe56c478527ade62 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } } }
I20250622 13:53:34.344682 5236 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2790345be1c04eddbe56c478527ade62 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } } }
W20250622 13:53:34.349169 5054 consensus_peers.cc:487] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 -> Peer 2790345be1c04eddbe56c478527ade62 (127.0.103.68:43953): Couldn't send request to peer 2790345be1c04eddbe56c478527ade62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50e27305bc8f4295a4a37be272baae9e. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:34.350179 4986 catalog_manager.cc:5582] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f reported cstate change: config changed from index -1 to 2, NON_VOTER 2790345be1c04eddbe56c478527ade62 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "041d5c7800494072b40a987b103573d7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } } }
I20250622 13:53:34.391183 5286 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2790345be1c04eddbe56c478527ade62 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } } }
I20250622 13:53:34.400296 5236 consensus_queue.cc:237] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.402458 5087 raft_consensus.cc:1273] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Refusing update from remote peer 4734b8dc14974411ad459aaf6a399c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:53:34.402868 5162 raft_consensus.cc:1273] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Refusing update from remote peer 4734b8dc14974411ad459aaf6a399c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:53:34.402987 5307 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:53:34.403373 5393 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:53:34.406387 5087 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2790345be1c04eddbe56c478527ade62 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } } }
I20250622 13:53:34.406461 5162 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2790345be1c04eddbe56c478527ade62 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } } }
W20250622 13:53:34.409549 5203 consensus_peers.cc:487] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f -> Peer 2790345be1c04eddbe56c478527ade62 (127.0.103.68:43953): Couldn't send request to peer 2790345be1c04eddbe56c478527ade62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4e25c6178ed498499bc9afc69eab8d5. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:34.417543 5406 ts_tablet_manager.cc:927] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: Initiating tablet copy from peer 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501)
I20250622 13:53:34.418145 5406 tablet_copy_client.cc:323] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:45501
I20250622 13:53:34.423327 5098 tablet_copy_service.cc:140] P 041d5c7800494072b40a987b103573d7: Received BeginTabletCopySession request for tablet 50e27305bc8f4295a4a37be272baae9e from peer 2790345be1c04eddbe56c478527ade62 ({username='slave'} at 127.0.0.1:60764)
I20250622 13:53:34.423472 5098 tablet_copy_service.cc:161] P 041d5c7800494072b40a987b103573d7: Beginning new tablet copy session on tablet 50e27305bc8f4295a4a37be272baae9e from peer 2790345be1c04eddbe56c478527ade62 at {username='slave'} at 127.0.0.1:60764: session id = 2790345be1c04eddbe56c478527ade62-50e27305bc8f4295a4a37be272baae9e
I20250622 13:53:34.425314 5098 tablet_copy_source_session.cc:215] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:34.426472 5406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e27305bc8f4295a4a37be272baae9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:34.430754 5406 tablet_copy_client.cc:806] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:34.430997 5406 tablet_copy_client.cc:670] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:34.432114 5392 raft_consensus.cc:991] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: : Instructing follower 1af80fe6712e4bd0b34e93169f5a0705 to start an election
I20250622 13:53:34.432273 5392 raft_consensus.cc:1079] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Signalling peer 1af80fe6712e4bd0b34e93169f5a0705 to start an election
I20250622 13:53:34.432726 5406 tablet_copy_client.cc:538] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:34.432902 5162 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e"
dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
from {username='slave'} at 127.0.0.1:34792
I20250622 13:53:34.433092 5162 raft_consensus.cc:491] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:34.433200 5162 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:34.435071 5162 raft_consensus.cc:513] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.435488 5406 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: Bootstrap starting.
I20250622 13:53:34.435850 5162 leader_election.cc:290] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 2 election: Requested vote from peers 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:34.436726 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4734b8dc14974411ad459aaf6a399c2f"
I20250622 13:53:34.436909 5236 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:34.437022 5087 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "041d5c7800494072b40a987b103573d7"
I20250622 13:53:34.437188 5087 raft_consensus.cc:3053] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:34.437263 5087 raft_consensus.cc:738] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 041d5c7800494072b40a987b103573d7, State: Running, Role: LEADER
I20250622 13:53:34.437568 5087 consensus_queue.cc:260] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.438015 5087 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:34.438133 4984 catalog_manager.cc:5582] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 reported cstate change: config changed from index -1 to 2, NON_VOTER 2790345be1c04eddbe56c478527ade62 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "4734b8dc14974411ad459aaf6a399c2f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } } }
I20250622 13:53:34.439792 5236 raft_consensus.cc:2466] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1af80fe6712e4bd0b34e93169f5a0705 in term 2.
I20250622 13:53:34.440634 5129 leader_election.cc:304] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af80fe6712e4bd0b34e93169f5a0705, 4734b8dc14974411ad459aaf6a399c2f; no voters:
I20250622 13:53:34.440711 5087 raft_consensus.cc:2466] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1af80fe6712e4bd0b34e93169f5a0705 in term 2.
I20250622 13:53:34.440959 5400 raft_consensus.cc:2802] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:34.442982 5400 raft_consensus.cc:695] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Becoming Leader. State: Replica: 1af80fe6712e4bd0b34e93169f5a0705, State: Running, Role: LEADER
I20250622 13:53:34.443255 5400 consensus_queue.cc:237] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.445070 5406 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:34.445389 5406 tablet_bootstrap.cc:492] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: Bootstrap complete.
I20250622 13:53:34.445611 5406 ts_tablet_manager.cc:1397] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.005s
I20250622 13:53:34.446174 5406 raft_consensus.cc:357] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.446314 5406 raft_consensus.cc:738] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2790345be1c04eddbe56c478527ade62, State: Initialized, Role: LEARNER
I20250622 13:53:34.446447 5406 consensus_queue.cc:260] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.447038 5406 ts_tablet_manager.cc:1428] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250622 13:53:34.447077 5380 heartbeater.cc:499] Master 127.0.103.126:32833 was elected leader, sending a full tablet report...
I20250622 13:53:34.447666 5098 tablet_copy_service.cc:342] P 041d5c7800494072b40a987b103573d7: Request end of tablet copy session 2790345be1c04eddbe56c478527ade62-50e27305bc8f4295a4a37be272baae9e received from {username='slave'} at 127.0.0.1:60764
I20250622 13:53:34.447791 5098 tablet_copy_service.cc:434] P 041d5c7800494072b40a987b103573d7: ending tablet copy session 2790345be1c04eddbe56c478527ade62-50e27305bc8f4295a4a37be272baae9e on tablet 50e27305bc8f4295a4a37be272baae9e with peer 2790345be1c04eddbe56c478527ade62
I20250622 13:53:34.499171 5393 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2790345be1c04eddbe56c478527ade62 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } } }
I20250622 13:53:34.502102 5406 ts_tablet_manager.cc:927] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: Initiating tablet copy from peer 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:34.502785 5406 tablet_copy_client.cc:323] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38299
I20250622 13:53:34.508244 5247 tablet_copy_service.cc:140] P 4734b8dc14974411ad459aaf6a399c2f: Received BeginTabletCopySession request for tablet c4e25c6178ed498499bc9afc69eab8d5 from peer 2790345be1c04eddbe56c478527ade62 ({username='slave'} at 127.0.0.1:57154)
I20250622 13:53:34.508412 5247 tablet_copy_service.cc:161] P 4734b8dc14974411ad459aaf6a399c2f: Beginning new tablet copy session on tablet c4e25c6178ed498499bc9afc69eab8d5 from peer 2790345be1c04eddbe56c478527ade62 at {username='slave'} at 127.0.0.1:57154: session id = 2790345be1c04eddbe56c478527ade62-c4e25c6178ed498499bc9afc69eab8d5
I20250622 13:53:34.510529 5247 tablet_copy_source_session.cc:215] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:34.511663 5406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e25c6178ed498499bc9afc69eab8d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:34.516816 5406 tablet_copy_client.cc:806] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:34.517082 5406 tablet_copy_client.cc:670] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:34.518590 5406 tablet_copy_client.cc:538] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:34.521406 5406 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: Bootstrap starting.
I20250622 13:53:34.527892 5406 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:34.528388 5406 tablet_bootstrap.cc:492] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: Bootstrap complete.
I20250622 13:53:34.528664 5406 ts_tablet_manager.cc:1397] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.008s
I20250622 13:53:34.529969 5406 raft_consensus.cc:357] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.530299 5406 raft_consensus.cc:738] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2790345be1c04eddbe56c478527ade62, State: Initialized, Role: LEARNER
I20250622 13:53:34.530580 5406 consensus_queue.cc:260] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.532141 5355 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 1 LEARNER]: Advancing to term 2
I20250622 13:53:34.533545 4986 catalog_manager.cc:5582] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 reported cstate change: term changed from 1 to 2, leader changed from 041d5c7800494072b40a987b103573d7 (127.0.103.65) to 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "1af80fe6712e4bd0b34e93169f5a0705" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:34.534541 5406 ts_tablet_manager.cc:1428] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250622 13:53:34.535187 5355 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 2 LEARNER]: Refusing update from remote peer 1af80fe6712e4bd0b34e93169f5a0705: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:53:34.535537 5247 tablet_copy_service.cc:342] P 4734b8dc14974411ad459aaf6a399c2f: Request end of tablet copy session 2790345be1c04eddbe56c478527ade62-c4e25c6178ed498499bc9afc69eab8d5 received from {username='slave'} at 127.0.0.1:57154
I20250622 13:53:34.536510 5247 tablet_copy_service.cc:434] P 4734b8dc14974411ad459aaf6a399c2f: ending tablet copy session 2790345be1c04eddbe56c478527ade62-c4e25c6178ed498499bc9afc69eab8d5 on tablet c4e25c6178ed498499bc9afc69eab8d5 with peer 2790345be1c04eddbe56c478527ade62
I20250622 13:53:34.536784 5400 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:53:34.540409 5411 raft_consensus.cc:1062] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: attempting to promote NON_VOTER 2790345be1c04eddbe56c478527ade62 to VOTER
W20250622 13:53:34.540551 5411 raft_consensus.cc:1066] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: Unable to promote non-voter 2790345be1c04eddbe56c478527ade62: Illegal state: Leader has not yet committed an operation in its own term
I20250622 13:53:34.548056 5236 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 2 FOLLOWER]: Refusing update from remote peer 1af80fe6712e4bd0b34e93169f5a0705: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:53:34.548614 5411 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:53:34.550458 5087 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 2 FOLLOWER]: Refusing update from remote peer 1af80fe6712e4bd0b34e93169f5a0705: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:53:34.551016 5411 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:53:34.569633 5162 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e"
dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34784
I20250622 13:53:34.569938 5162 raft_consensus.cc:604] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Received request to transfer leadership
I20250622 13:53:34.572999 5162 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e"
dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34784
I20250622 13:53:34.573197 5162 raft_consensus.cc:604] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Received request to transfer leadership
I20250622 13:53:34.574577 5162 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e"
dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34784
I20250622 13:53:34.574749 5162 raft_consensus.cc:604] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Received request to transfer leadership
W20250622 13:53:34.575205 5034 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 50e27305bc8f4295a4a37be272baae9e already in progress
W20250622 13:53:34.575284 5034 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 50e27305bc8f4295a4a37be272baae9e already in progress
I20250622 13:53:34.587097 5355 raft_consensus.cc:1215] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250622 13:53:34.671512 5400 raft_consensus.cc:991] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: : Instructing follower 041d5c7800494072b40a987b103573d7 to start an election
I20250622 13:53:34.671737 5400 raft_consensus.cc:1079] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Signalling peer 041d5c7800494072b40a987b103573d7 to start an election
I20250622 13:53:34.672299 5088 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e"
dest_uuid: "041d5c7800494072b40a987b103573d7"
from {username='slave'} at 127.0.0.1:60736
I20250622 13:53:34.672488 5088 raft_consensus.cc:491] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:34.672626 5088 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:34.675266 5088 raft_consensus.cc:513] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.676003 5088 leader_election.cc:290] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 3 election: Requested vote from peers 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:34.676388 5162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
I20250622 13:53:34.676388 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e27305bc8f4295a4a37be272baae9e" candidate_uuid: "041d5c7800494072b40a987b103573d7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "4734b8dc14974411ad459aaf6a399c2f"
I20250622 13:53:34.676620 5162 raft_consensus.cc:3053] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:53:34.676640 5236 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:34.676745 5162 raft_consensus.cc:738] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1af80fe6712e4bd0b34e93169f5a0705, State: Running, Role: LEADER
I20250622 13:53:34.677043 5162 consensus_queue.cc:260] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.677469 5162 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:53:34.679358 5236 raft_consensus.cc:2466] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 3.
I20250622 13:53:34.679759 5054 leader_election.cc:304] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d5c7800494072b40a987b103573d7, 4734b8dc14974411ad459aaf6a399c2f; no voters:
I20250622 13:53:34.680055 5302 raft_consensus.cc:2802] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:53:34.680217 5302 raft_consensus.cc:695] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 3 LEADER]: Becoming Leader. State: Replica: 041d5c7800494072b40a987b103573d7, State: Running, Role: LEADER
I20250622 13:53:34.680516 5302 consensus_queue.cc:237] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } }
I20250622 13:53:34.683153 4983 catalog_manager.cc:5582] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 reported cstate change: term changed from 2 to 3, leader changed from 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66) to 041d5c7800494072b40a987b103573d7 (127.0.103.65). New cstate: current_term: 3 leader_uuid: "041d5c7800494072b40a987b103573d7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:34.702064 5162 raft_consensus.cc:2466] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041d5c7800494072b40a987b103573d7 in term 3.
I20250622 13:53:34.719480 5393 raft_consensus.cc:1062] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f: attempting to promote NON_VOTER 2790345be1c04eddbe56c478527ade62 to VOTER
I20250622 13:53:34.720171 5393 consensus_queue.cc:237] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } }
I20250622 13:53:34.721923 5162 raft_consensus.cc:1273] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Refusing update from remote peer 4734b8dc14974411ad459aaf6a399c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:53:34.721949 5355 raft_consensus.cc:1273] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 LEARNER]: Refusing update from remote peer 4734b8dc14974411ad459aaf6a399c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:53:34.722232 5088 raft_consensus.cc:1273] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Refusing update from remote peer 4734b8dc14974411ad459aaf6a399c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:53:34.722591 5424 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:53:34.722860 5404 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:53:34.723129 5307 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:53:34.726392 5355 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.726846 5162 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.731150 4984 catalog_manager.cc:5582] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 reported cstate change: config changed from index 2 to 3, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4734b8dc14974411ad459aaf6a399c2f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.754801 5307 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.758450 5088 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.774760 5162 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 3 FOLLOWER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250622 13:53:34.775311 5302 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:53:34.779778 5237 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 3 FOLLOWER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250622 13:53:34.780269 5302 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:53:34.781410 5355 raft_consensus.cc:3058] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 2 LEARNER]: Advancing to term 3
I20250622 13:53:34.783694 5355 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 3 LEARNER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250622 13:53:34.784179 5302 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:53:34.868073 5237 consensus_queue.cc:237] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } }
I20250622 13:53:34.870216 5355 raft_consensus.cc:1273] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 FOLLOWER]: Refusing update from remote peer 4734b8dc14974411ad459aaf6a399c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250622 13:53:34.870582 5162 raft_consensus.cc:1273] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Refusing update from remote peer 4734b8dc14974411ad459aaf6a399c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250622 13:53:34.870772 5307 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:53:34.871204 5424 consensus_queue.cc:1035] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:53:34.873989 5162 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 041d5c7800494072b40a987b103573d7 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.874084 5355 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 041d5c7800494072b40a987b103573d7 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.877724 4985 catalog_manager.cc:5582] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 reported cstate change: config changed from index 3 to 4, VOTER 041d5c7800494072b40a987b103573d7 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "4734b8dc14974411ad459aaf6a399c2f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.891450 5424 raft_consensus.cc:2953] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 041d5c7800494072b40a987b103573d7 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.892383 5076 tablet_service.cc:1515] Processing DeleteTablet for tablet c4e25c6178ed498499bc9afc69eab8d5 with delete_type TABLET_DATA_TOMBSTONED (TS 041d5c7800494072b40a987b103573d7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60720
I20250622 13:53:34.893188 5426 tablet_replica.cc:331] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: stopping tablet replica
I20250622 13:53:34.893424 5426 raft_consensus.cc:2241] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:34.893684 5426 raft_consensus.cc:2270] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:34.894469 5426 ts_tablet_manager.cc:1905] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:34.895102 4973 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c4e25c6178ed498499bc9afc69eab8d5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:34.899482 5426 ts_tablet_manager.cc:1918] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250622 13:53:34.899633 5426 log.cc:1199] T c4e25c6178ed498499bc9afc69eab8d5 P 041d5c7800494072b40a987b103573d7: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-0-root/wals/c4e25c6178ed498499bc9afc69eab8d5
I20250622 13:53:34.900342 4972 catalog_manager.cc:4928] TS 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501): tablet c4e25c6178ed498499bc9afc69eab8d5 (table test-workload [id=a990b1eb84604eaeaff0578e61433b84]) successfully deleted
I20250622 13:53:34.902575 5302 raft_consensus.cc:1062] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: attempting to promote NON_VOTER 2790345be1c04eddbe56c478527ade62 to VOTER
I20250622 13:53:34.903335 5302 consensus_queue.cc:237] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } }
I20250622 13:53:34.905787 5236 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 3 FOLLOWER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250622 13:53:34.905787 5355 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 3 LEARNER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250622 13:53:34.906026 5162 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 3 FOLLOWER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250622 13:53:34.906467 5301 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:53:34.906752 5302 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:53:34.907029 5427 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:53:34.910084 5355 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.910084 5236 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:34.914940 4985 catalog_manager.cc:5582] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f reported cstate change: config changed from index 2 to 5, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "041d5c7800494072b40a987b103573d7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:35.028190 5162 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:35.126392 5392 raft_consensus.cc:1025] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 3 LEADER]: attempt to promote peer 2790345be1c04eddbe56c478527ade62: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250622 13:53:35.210530 5427 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 2790345be1c04eddbe56c478527ade62 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } attrs { replace: true } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:35.215853 5088 consensus_queue.cc:237] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } }
I20250622 13:53:35.217291 5236 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 3 FOLLOWER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250622 13:53:35.217386 5355 raft_consensus.cc:1273] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 3 FOLLOWER]: Refusing update from remote peer 041d5c7800494072b40a987b103573d7: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250622 13:53:35.217792 5392 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:53:35.217983 5301 consensus_queue.cc:1035] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:53:35.220257 5236 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:35.220405 5355 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:35.224251 4983 catalog_manager.cc:5582] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 reported cstate change: config changed from index 5 to 6, VOTER 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66) evicted. New cstate: current_term: 3 leader_uuid: "041d5c7800494072b40a987b103573d7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:35.245832 5151 tablet_service.cc:1515] Processing DeleteTablet for tablet 50e27305bc8f4295a4a37be272baae9e with delete_type TABLET_DATA_TOMBSTONED (TS 1af80fe6712e4bd0b34e93169f5a0705 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34784
I20250622 13:53:35.246436 5428 tablet_replica.cc:331] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: stopping tablet replica
I20250622 13:53:35.246599 5428 raft_consensus.cc:2241] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:35.246796 5428 raft_consensus.cc:2270] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:35.247463 5428 ts_tablet_manager.cc:1905] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:53:35.251617 5428 ts_tablet_manager.cc:1918] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20250622 13:53:35.251708 5428 log.cc:1199] T 50e27305bc8f4295a4a37be272baae9e P 1af80fe6712e4bd0b34e93169f5a0705: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750600281110842-413-0/minicluster-data/ts-1-root/wals/50e27305bc8f4295a4a37be272baae9e
I20250622 13:53:35.252363 4972 catalog_manager.cc:4928] TS 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66:46717): tablet 50e27305bc8f4295a4a37be272baae9e (table test-workload [id=a990b1eb84604eaeaff0578e61433b84]) successfully deleted
I20250622 13:53:35.508265 5392 raft_consensus.cc:2953] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } } peers { permanent_uuid: "2790345be1c04eddbe56c478527ade62" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 43953 } attrs { promote: false } } }
I20250622 13:53:35.511198 4972 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 50e27305bc8f4295a4a37be272baae9e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:53:36.342559 5039 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:36.344271 5088 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cfe0044d7bce4e93813f9883aece8502"
dest_uuid: "041d5c7800494072b40a987b103573d7"
mode: GRACEFUL
new_leader_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
from {username='slave'} at 127.0.0.1:60762
I20250622 13:53:36.344449 5088 raft_consensus.cc:604] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Received request to transfer leadership to TS 1af80fe6712e4bd0b34e93169f5a0705
I20250622 13:53:36.344939 5039 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250622 13:53:36.345054 5039 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:36.456357 5427 raft_consensus.cc:991] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7: : Instructing follower 1af80fe6712e4bd0b34e93169f5a0705 to start an election
I20250622 13:53:36.456602 5427 raft_consensus.cc:1079] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Signalling peer 1af80fe6712e4bd0b34e93169f5a0705 to start an election
I20250622 13:53:36.457450 5162 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "cfe0044d7bce4e93813f9883aece8502"
dest_uuid: "1af80fe6712e4bd0b34e93169f5a0705"
from {username='slave'} at 127.0.0.1:34792
I20250622 13:53:36.457648 5162 raft_consensus.cc:491] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:53:36.457755 5162 raft_consensus.cc:3058] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:36.459692 5162 raft_consensus.cc:513] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:36.460381 5162 leader_election.cc:290] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 2 election: Requested vote from peers 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501), 4734b8dc14974411ad459aaf6a399c2f (127.0.103.67:38299)
I20250622 13:53:36.460783 5088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfe0044d7bce4e93813f9883aece8502" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "041d5c7800494072b40a987b103573d7"
I20250622 13:53:36.460813 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfe0044d7bce4e93813f9883aece8502" candidate_uuid: "1af80fe6712e4bd0b34e93169f5a0705" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4734b8dc14974411ad459aaf6a399c2f"
I20250622 13:53:36.460963 5236 raft_consensus.cc:3058] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:36.460963 5088 raft_consensus.cc:3053] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:53:36.461058 5088 raft_consensus.cc:738] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 041d5c7800494072b40a987b103573d7, State: Running, Role: LEADER
I20250622 13:53:36.461299 5088 consensus_queue.cc:260] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:36.461635 5088 raft_consensus.cc:3058] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:36.463908 5236 raft_consensus.cc:2466] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1af80fe6712e4bd0b34e93169f5a0705 in term 2.
I20250622 13:53:36.464205 5088 raft_consensus.cc:2466] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1af80fe6712e4bd0b34e93169f5a0705 in term 2.
I20250622 13:53:36.464403 5129 leader_election.cc:304] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af80fe6712e4bd0b34e93169f5a0705, 4734b8dc14974411ad459aaf6a399c2f; no voters:
I20250622 13:53:36.464956 5430 raft_consensus.cc:2802] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:36.465253 5430 raft_consensus.cc:695] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Becoming Leader. State: Replica: 1af80fe6712e4bd0b34e93169f5a0705, State: Running, Role: LEADER
I20250622 13:53:36.465523 5430 consensus_queue.cc:237] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } }
I20250622 13:53:36.467880 4986 catalog_manager.cc:5582] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 reported cstate change: term changed from 1 to 2, leader changed from 041d5c7800494072b40a987b103573d7 (127.0.103.65) to 1af80fe6712e4bd0b34e93169f5a0705 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "1af80fe6712e4bd0b34e93169f5a0705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af80fe6712e4bd0b34e93169f5a0705" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:36.542135 5236 raft_consensus.cc:1273] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 2 FOLLOWER]: Refusing update from remote peer 1af80fe6712e4bd0b34e93169f5a0705: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:53:36.542598 5430 consensus_queue.cc:1035] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4734b8dc14974411ad459aaf6a399c2f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:53:36.554378 5088 raft_consensus.cc:1273] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 2 FOLLOWER]: Refusing update from remote peer 1af80fe6712e4bd0b34e93169f5a0705: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:53:36.555056 5430 consensus_queue.cc:1035] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "041d5c7800494072b40a987b103573d7" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:53:38.345387 5039 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:38.346083 5039 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:38.346192 5039 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:40.346670 5039 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:40.347405 5039 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:40.347509 5039 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:42.347941 5039 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:42.348654 5039 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:42.348768 5039 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:42.833977 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
W20250622 13:53:42.856088 5128 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20250622 13:53:42.857573 5128 consensus_peers.cc:487] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 -> Peer 041d5c7800494072b40a987b103573d7 (127.0.103.65:45501): Couldn't send request to peer 041d5c7800494072b40a987b103573d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:42.861604 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:42.861855 413 tablet_replica.cc:331] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7: stopping tablet replica
I20250622 13:53:42.862039 413 raft_consensus.cc:2241] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 3 LEADER]: Raft consensus shutting down.
I20250622 13:53:42.862331 413 raft_consensus.cc:2270] T 50e27305bc8f4295a4a37be272baae9e P 041d5c7800494072b40a987b103573d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.862888 413 tablet_replica.cc:331] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7: stopping tablet replica
I20250622 13:53:42.863040 413 raft_consensus.cc:2241] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:42.863185 413 raft_consensus.cc:2270] T cfe0044d7bce4e93813f9883aece8502 P 041d5c7800494072b40a987b103573d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.894080 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:53:42.897532 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:53:42.906498 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:42.906766 413 tablet_replica.cc:331] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705: stopping tablet replica
I20250622 13:53:42.906922 413 raft_consensus.cc:2241] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:42.907176 413 raft_consensus.cc:2270] T c4e25c6178ed498499bc9afc69eab8d5 P 1af80fe6712e4bd0b34e93169f5a0705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.907824 413 tablet_replica.cc:331] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705: stopping tablet replica
I20250622 13:53:42.907967 413 raft_consensus.cc:2241] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:53:42.908293 413 raft_consensus.cc:2270] T cfe0044d7bce4e93813f9883aece8502 P 1af80fe6712e4bd0b34e93169f5a0705 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.923126 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:53:42.927273 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:53:42.936969 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:42.937297 413 tablet_replica.cc:331] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f: stopping tablet replica
I20250622 13:53:42.937489 413 raft_consensus.cc:2241] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:42.937713 413 raft_consensus.cc:2270] T 50e27305bc8f4295a4a37be272baae9e P 4734b8dc14974411ad459aaf6a399c2f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.938288 413 tablet_replica.cc:331] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f: stopping tablet replica
I20250622 13:53:42.938422 413 raft_consensus.cc:2241] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:53:42.938680 413 raft_consensus.cc:2270] T c4e25c6178ed498499bc9afc69eab8d5 P 4734b8dc14974411ad459aaf6a399c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.939229 413 tablet_replica.cc:331] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f: stopping tablet replica
I20250622 13:53:42.939368 413 raft_consensus.cc:2241] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:42.939587 413 raft_consensus.cc:2270] T cfe0044d7bce4e93813f9883aece8502 P 4734b8dc14974411ad459aaf6a399c2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.953853 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:53:42.957471 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250622 13:53:42.966951 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:42.967602 413 tablet_replica.cc:331] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62: stopping tablet replica
I20250622 13:53:42.967939 413 raft_consensus.cc:2241] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:42.968250 413 raft_consensus.cc:2270] T c4e25c6178ed498499bc9afc69eab8d5 P 2790345be1c04eddbe56c478527ade62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.968871 413 tablet_replica.cc:331] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62: stopping tablet replica
I20250622 13:53:42.969019 413 raft_consensus.cc:2241] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:42.969264 413 raft_consensus.cc:2270] T 50e27305bc8f4295a4a37be272baae9e P 2790345be1c04eddbe56c478527ade62 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:42.983160 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250622 13:53:42.986398 413 master.cc:561] Master@127.0.103.126:32833 shutting down...
I20250622 13:53:43.010752 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:53:43.011039 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:43.011149 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3bda0cf81e724234a2793350f7a3f495: stopping tablet replica
I20250622 13:53:43.024669 413 master.cc:583] Master@127.0.103.126:32833 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10770 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20250622 13:53:43.037721 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:44135
I20250622 13:53:43.038203 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:43.040678 5446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:43.040688 5447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:43.040678 5449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:43.040930 413 server_base.cc:1048] running on GCE node
I20250622 13:53:43.041577 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:43.041639 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:43.041673 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600423041668 us; error 0 us; skew 500 ppm
I20250622 13:53:43.041855 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:43.047114 413 webserver.cc:469] Webserver started at http://127.0.103.126:45139/ using document root <none> and password file <none>
I20250622 13:53:43.047348 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:43.047425 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:43.047551 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:43.048200 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "376e98a8e8924773b022dbfb36da272a"
format_stamp: "Formatted at 2025-06-22 13:53:43 on dist-test-slave-tmv0"
I20250622 13:53:43.050963 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:43.052502 5454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.052891 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:53:43.052975 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "376e98a8e8924773b022dbfb36da272a"
format_stamp: "Formatted at 2025-06-22 13:53:43 on dist-test-slave-tmv0"
I20250622 13:53:43.053104 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:43.064656 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:43.065182 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:43.077872 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:44135
I20250622 13:53:43.077908 5505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:44135 every 8 connection(s)
I20250622 13:53:43.079242 5506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:43.083397 5506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a: Bootstrap starting.
I20250622 13:53:43.084689 5506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:43.086354 5506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a: No bootstrap required, opened a new log
I20250622 13:53:43.087049 5506 raft_consensus.cc:357] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376e98a8e8924773b022dbfb36da272a" member_type: VOTER }
I20250622 13:53:43.087184 5506 raft_consensus.cc:383] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:43.087258 5506 raft_consensus.cc:738] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e98a8e8924773b022dbfb36da272a, State: Initialized, Role: FOLLOWER
I20250622 13:53:43.087493 5506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376e98a8e8924773b022dbfb36da272a" member_type: VOTER }
I20250622 13:53:43.087625 5506 raft_consensus.cc:397] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:53:43.087695 5506 raft_consensus.cc:491] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:53:43.087790 5506 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:43.090282 5506 raft_consensus.cc:513] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376e98a8e8924773b022dbfb36da272a" member_type: VOTER }
I20250622 13:53:43.090503 5506 leader_election.cc:304] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 376e98a8e8924773b022dbfb36da272a; no voters:
I20250622 13:53:43.090926 5506 leader_election.cc:290] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:53:43.091070 5509 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:43.091491 5509 raft_consensus.cc:695] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 1 LEADER]: Becoming Leader. State: Replica: 376e98a8e8924773b022dbfb36da272a, State: Running, Role: LEADER
I20250622 13:53:43.091710 5509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376e98a8e8924773b022dbfb36da272a" member_type: VOTER }
I20250622 13:53:43.091816 5506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:53:43.092464 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 376e98a8e8924773b022dbfb36da272a. Latest consensus state: current_term: 1 leader_uuid: "376e98a8e8924773b022dbfb36da272a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376e98a8e8924773b022dbfb36da272a" member_type: VOTER } }
I20250622 13:53:43.092613 5510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "376e98a8e8924773b022dbfb36da272a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376e98a8e8924773b022dbfb36da272a" member_type: VOTER } }
I20250622 13:53:43.092689 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [sys.catalog]: This master's current role is: LEADER
I20250622 13:53:43.092862 5510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [sys.catalog]: This master's current role is: LEADER
I20250622 13:53:43.094273 5514 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:53:43.095592 5514 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:53:43.097395 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:53:43.098466 5514 catalog_manager.cc:1349] Generated new cluster ID: 578a2ebe172d4e55a704faf7a9fe83a5
I20250622 13:53:43.098541 5514 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:53:43.112074 5514 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:53:43.112820 5514 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:53:43.118917 5514 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a: Generated new TSK 0
I20250622 13:53:43.119138 5514 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:53:43.130271 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:43.132829 5527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:43.133062 5528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:43.133639 5530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:43.134215 413 server_base.cc:1048] running on GCE node
I20250622 13:53:43.134462 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:43.134524 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:43.134583 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600423134564 us; error 0 us; skew 500 ppm
I20250622 13:53:43.134740 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:43.135972 413 webserver.cc:469] Webserver started at http://127.0.103.65:45279/ using document root <none> and password file <none>
I20250622 13:53:43.136204 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:43.136282 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:43.136431 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:43.137100 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "b529cf0a8ccf4100bb0b9c07baead16d"
format_stamp: "Formatted at 2025-06-22 13:53:43 on dist-test-slave-tmv0"
I20250622 13:53:43.139715 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20250622 13:53:43.141466 5535 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.141844 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:53:43.141966 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "b529cf0a8ccf4100bb0b9c07baead16d"
format_stamp: "Formatted at 2025-06-22 13:53:43 on dist-test-slave-tmv0"
I20250622 13:53:43.142095 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:43.154199 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:43.154819 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:43.155509 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:43.156405 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:43.156482 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.156594 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:43.156656 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.171202 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39899
I20250622 13:53:43.171241 5597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39899 every 8 connection(s)
I20250622 13:53:43.172821 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:43.175618 5603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:43.175751 5602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:43.177428 5605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:43.179258 413 server_base.cc:1048] running on GCE node
I20250622 13:53:43.179590 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:43.179682 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:43.179738 5598 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44135
I20250622 13:53:43.179783 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600423179761 us; error 0 us; skew 500 ppm
I20250622 13:53:43.179906 5598 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:43.179956 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:43.180315 5598 heartbeater.cc:507] Master 127.0.103.126:44135 requested a full tablet report, sending...
I20250622 13:53:43.181347 5471 ts_manager.cc:194] Registered new tserver with Master: b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899)
I20250622 13:53:43.181484 413 webserver.cc:469] Webserver started at http://127.0.103.66:38485/ using document root <none> and password file <none>
I20250622 13:53:43.181720 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:43.181833 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:43.181975 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:43.182770 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "bf17379793784cd3a8862e6688948fc8"
format_stamp: "Formatted at 2025-06-22 13:53:43 on dist-test-slave-tmv0"
I20250622 13:53:43.182912 5471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36380
I20250622 13:53:43.185240 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:43.186712 5610 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.187063 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:53:43.187176 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "bf17379793784cd3a8862e6688948fc8"
format_stamp: "Formatted at 2025-06-22 13:53:43 on dist-test-slave-tmv0"
I20250622 13:53:43.187289 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:43.196369 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:43.196866 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:43.197491 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:43.198333 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:43.198405 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.198506 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:43.198565 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.210662 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:36787
I20250622 13:53:43.210685 5672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:36787 every 8 connection(s)
I20250622 13:53:43.212213 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:43.215159 5676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:43.215477 5679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:43.215833 5677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:43.219233 5673 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44135
I20250622 13:53:43.219430 5673 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:43.219971 5673 heartbeater.cc:507] Master 127.0.103.126:44135 requested a full tablet report, sending...
I20250622 13:53:43.220961 5471 ts_manager.cc:194] Registered new tserver with Master: bf17379793784cd3a8862e6688948fc8 (127.0.103.66:36787)
I20250622 13:53:43.221881 413 server_base.cc:1048] running on GCE node
I20250622 13:53:43.221956 5471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36382
I20250622 13:53:43.222226 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:43.222322 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:43.222390 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600423222365 us; error 0 us; skew 500 ppm
I20250622 13:53:43.222560 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:43.223843 413 webserver.cc:469] Webserver started at http://127.0.103.67:38535/ using document root <none> and password file <none>
I20250622 13:53:43.224056 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:43.224145 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:43.224277 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:43.224877 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "6f75feff0d924a178373e475de012366"
format_stamp: "Formatted at 2025-06-22 13:53:43 on dist-test-slave-tmv0"
I20250622 13:53:43.227345 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:43.229252 5684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.229655 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:53:43.229779 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "6f75feff0d924a178373e475de012366"
format_stamp: "Formatted at 2025-06-22 13:53:43 on dist-test-slave-tmv0"
I20250622 13:53:43.229899 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:43.239482 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:43.239993 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:43.240535 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:43.241336 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:43.241456 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.241564 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:43.241631 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:43.254967 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:36201
I20250622 13:53:43.255016 5746 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:36201 every 8 connection(s)
I20250622 13:53:43.261379 5747 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44135
I20250622 13:53:43.261535 5747 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:43.261890 5747 heartbeater.cc:507] Master 127.0.103.126:44135 requested a full tablet report, sending...
I20250622 13:53:43.262676 5471 ts_manager.cc:194] Registered new tserver with Master: 6f75feff0d924a178373e475de012366 (127.0.103.67:36201)
I20250622 13:53:43.262696 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006864477s
I20250622 13:53:43.263720 5471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36390
I20250622 13:53:44.184216 5598 heartbeater.cc:499] Master 127.0.103.126:44135 was elected leader, sending a full tablet report...
I20250622 13:53:44.223414 5673 heartbeater.cc:499] Master 127.0.103.126:44135 was elected leader, sending a full tablet report...
I20250622 13:53:44.264833 5747 heartbeater.cc:499] Master 127.0.103.126:44135 was elected leader, sending a full tablet report...
I20250622 13:53:44.290874 413 test_util.cc:276] Using random seed: 1819212310
I20250622 13:53:44.302394 5471 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36400:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:53:44.302942 5471 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:53:44.318603 5563 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac76cf36fb84da78b209c977087daac (DEFAULT_TABLE table=test-workload [id=35007d00f5de47b9bb0e8301a19c043c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:53:44.318961 5712 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac76cf36fb84da78b209c977087daac (DEFAULT_TABLE table=test-workload [id=35007d00f5de47b9bb0e8301a19c043c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:53:44.319128 5563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac76cf36fb84da78b209c977087daac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:44.319329 5712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac76cf36fb84da78b209c977087daac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:44.321756 5638 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac76cf36fb84da78b209c977087daac (DEFAULT_TABLE table=test-workload [id=35007d00f5de47b9bb0e8301a19c043c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:53:44.322145 5638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac76cf36fb84da78b209c977087daac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:44.325570 5767 tablet_bootstrap.cc:492] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366: Bootstrap starting.
I20250622 13:53:44.327085 5767 tablet_bootstrap.cc:654] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:44.328493 5768 tablet_bootstrap.cc:492] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d: Bootstrap starting.
I20250622 13:53:44.328969 5767 tablet_bootstrap.cc:492] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366: No bootstrap required, opened a new log
I20250622 13:53:44.329142 5767 ts_tablet_manager.cc:1397] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:53:44.329416 5770 tablet_bootstrap.cc:492] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8: Bootstrap starting.
I20250622 13:53:44.329969 5767 raft_consensus.cc:357] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.330127 5767 raft_consensus.cc:383] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:44.330196 5767 raft_consensus.cc:738] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f75feff0d924a178373e475de012366, State: Initialized, Role: FOLLOWER
I20250622 13:53:44.330420 5767 consensus_queue.cc:260] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.331110 5770 tablet_bootstrap.cc:654] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:44.331110 5768 tablet_bootstrap.cc:654] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:44.331809 5767 ts_tablet_manager.cc:1428] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250622 13:53:44.333191 5768 tablet_bootstrap.cc:492] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d: No bootstrap required, opened a new log
I20250622 13:53:44.333385 5768 ts_tablet_manager.cc:1397] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250622 13:53:44.333588 5770 tablet_bootstrap.cc:492] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8: No bootstrap required, opened a new log
I20250622 13:53:44.333726 5770 ts_tablet_manager.cc:1397] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:53:44.334439 5768 raft_consensus.cc:357] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.334622 5770 raft_consensus.cc:357] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.334772 5768 raft_consensus.cc:383] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:44.334816 5770 raft_consensus.cc:383] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:44.335027 5768 raft_consensus.cc:738] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b529cf0a8ccf4100bb0b9c07baead16d, State: Initialized, Role: FOLLOWER
I20250622 13:53:44.335098 5770 raft_consensus.cc:738] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf17379793784cd3a8862e6688948fc8, State: Initialized, Role: FOLLOWER
I20250622 13:53:44.335346 5768 consensus_queue.cc:260] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.335345 5770 consensus_queue.cc:260] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.336643 5770 ts_tablet_manager.cc:1428] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250622 13:53:44.336934 5768 ts_tablet_manager.cc:1428] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250622 13:53:44.398864 5774 raft_consensus.cc:491] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:44.399124 5774 raft_consensus.cc:513] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.399962 5774 leader_election.cc:290] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f75feff0d924a178373e475de012366 (127.0.103.67:36201), bf17379793784cd3a8862e6688948fc8 (127.0.103.66:36787)
I20250622 13:53:44.406005 5722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac76cf36fb84da78b209c977087daac" candidate_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f75feff0d924a178373e475de012366" is_pre_election: true
I20250622 13:53:44.406244 5722 raft_consensus.cc:2466] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b529cf0a8ccf4100bb0b9c07baead16d in term 0.
I20250622 13:53:44.406252 5648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac76cf36fb84da78b209c977087daac" candidate_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf17379793784cd3a8862e6688948fc8" is_pre_election: true
I20250622 13:53:44.406484 5648 raft_consensus.cc:2466] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b529cf0a8ccf4100bb0b9c07baead16d in term 0.
I20250622 13:53:44.406706 5537 leader_election.cc:304] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f75feff0d924a178373e475de012366, b529cf0a8ccf4100bb0b9c07baead16d; no voters:
I20250622 13:53:44.406937 5774 raft_consensus.cc:2802] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:44.407023 5774 raft_consensus.cc:491] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:44.407104 5774 raft_consensus.cc:3058] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:44.408900 5774 raft_consensus.cc:513] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.409478 5774 leader_election.cc:290] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [CANDIDATE]: Term 1 election: Requested vote from peers 6f75feff0d924a178373e475de012366 (127.0.103.67:36201), bf17379793784cd3a8862e6688948fc8 (127.0.103.66:36787)
I20250622 13:53:44.409832 5722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac76cf36fb84da78b209c977087daac" candidate_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f75feff0d924a178373e475de012366"
I20250622 13:53:44.409875 5648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac76cf36fb84da78b209c977087daac" candidate_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf17379793784cd3a8862e6688948fc8"
I20250622 13:53:44.410013 5722 raft_consensus.cc:3058] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:44.410074 5648 raft_consensus.cc:3058] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:44.411957 5722 raft_consensus.cc:2466] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b529cf0a8ccf4100bb0b9c07baead16d in term 1.
I20250622 13:53:44.411998 5648 raft_consensus.cc:2466] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b529cf0a8ccf4100bb0b9c07baead16d in term 1.
I20250622 13:53:44.412317 5537 leader_election.cc:304] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f75feff0d924a178373e475de012366, b529cf0a8ccf4100bb0b9c07baead16d; no voters:
I20250622 13:53:44.412571 5774 raft_consensus.cc:2802] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:44.412955 5774 raft_consensus.cc:695] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 1 LEADER]: Becoming Leader. State: Replica: b529cf0a8ccf4100bb0b9c07baead16d, State: Running, Role: LEADER
I20250622 13:53:44.413203 5774 consensus_queue.cc:237] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:44.415500 5471 catalog_manager.cc:5582] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d reported cstate change: term changed from 0 to 1, leader changed from <none> to b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65). New cstate: current_term: 1 leader_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:44.435029 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:53:44.444231 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:44.444675 413 tablet_replica.cc:331] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d: stopping tablet replica
I20250622 13:53:44.444909 413 raft_consensus.cc:2241] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:53:44.445223 413 pending_rounds.cc:70] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d: Trying to abort 1 pending ops.
I20250622 13:53:44.445453 413 pending_rounds.cc:77] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459338398552064 op_type: NO_OP noop_request { }
I20250622 13:53:44.445611 413 raft_consensus.cc:2887] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:53:44.445742 413 raft_consensus.cc:2270] T 9ac76cf36fb84da78b209c977087daac P b529cf0a8ccf4100bb0b9c07baead16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:44.460405 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:53:45.097267 5524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:45.097631 5524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:45.097704 5524 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:46.263695 5780 raft_consensus.cc:491] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:46.263876 5780 raft_consensus.cc:513] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:46.264669 5780 leader_election.cc:290] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f75feff0d924a178373e475de012366 (127.0.103.67:36201), b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899)
W20250622 13:53:46.267401 5612 leader_election.cc:336] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899): Network error: Client connection negotiation failed: client connection to 127.0.103.65:39899: connect: Connection refused (error 111)
I20250622 13:53:46.270920 5722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac76cf36fb84da78b209c977087daac" candidate_uuid: "bf17379793784cd3a8862e6688948fc8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f75feff0d924a178373e475de012366" is_pre_election: true
I20250622 13:53:46.271129 5722 raft_consensus.cc:2466] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf17379793784cd3a8862e6688948fc8 in term 1.
I20250622 13:53:46.271526 5612 leader_election.cc:304] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f75feff0d924a178373e475de012366, bf17379793784cd3a8862e6688948fc8; no voters: b529cf0a8ccf4100bb0b9c07baead16d
I20250622 13:53:46.271772 5780 raft_consensus.cc:2802] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:53:46.271847 5780 raft_consensus.cc:491] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:46.271919 5780 raft_consensus.cc:3058] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:46.273818 5780 raft_consensus.cc:513] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:46.274322 5780 leader_election.cc:290] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [CANDIDATE]: Term 2 election: Requested vote from peers 6f75feff0d924a178373e475de012366 (127.0.103.67:36201), b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899)
I20250622 13:53:46.274736 5722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ac76cf36fb84da78b209c977087daac" candidate_uuid: "bf17379793784cd3a8862e6688948fc8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f75feff0d924a178373e475de012366"
I20250622 13:53:46.274912 5722 raft_consensus.cc:3058] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 1 FOLLOWER]: Advancing to term 2
W20250622 13:53:46.275586 5612 leader_election.cc:336] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899): Network error: Client connection negotiation failed: client connection to 127.0.103.65:39899: connect: Connection refused (error 111)
I20250622 13:53:46.277387 5722 raft_consensus.cc:2466] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf17379793784cd3a8862e6688948fc8 in term 2.
I20250622 13:53:46.277720 5612 leader_election.cc:304] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f75feff0d924a178373e475de012366, bf17379793784cd3a8862e6688948fc8; no voters: b529cf0a8ccf4100bb0b9c07baead16d
I20250622 13:53:46.277949 5780 raft_consensus.cc:2802] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:46.278339 5780 raft_consensus.cc:695] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 2 LEADER]: Becoming Leader. State: Replica: bf17379793784cd3a8862e6688948fc8, State: Running, Role: LEADER
I20250622 13:53:46.278595 5780 consensus_queue.cc:237] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } }
I20250622 13:53:46.280772 5471 catalog_manager.cc:5582] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 reported cstate change: term changed from 1 to 2, leader changed from b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65) to bf17379793784cd3a8862e6688948fc8 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "bf17379793784cd3a8862e6688948fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b529cf0a8ccf4100bb0b9c07baead16d" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf17379793784cd3a8862e6688948fc8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36787 } health_report { overall_health: HEALTHY } } }
W20250622 13:53:46.763988 5612 consensus_peers.cc:487] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 -> Peer b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899): Couldn't send request to peer b529cf0a8ccf4100bb0b9c07baead16d. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:46.766332 5722 raft_consensus.cc:1273] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 2 FOLLOWER]: Refusing update from remote peer bf17379793784cd3a8862e6688948fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250622 13:53:46.766929 5780 consensus_queue.cc:1035] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f75feff0d924a178373e475de012366" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:53:47.098141 5524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:47.098585 5524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:47.098696 5524 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250622 13:53:48.309242 5612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39899: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20250622 13:53:49.099045 5524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:49.099403 5524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:49.099471 5524 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250622 13:53:49.342366 5612 consensus_peers.cc:487] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 -> Peer b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899): Couldn't send request to peer b529cf0a8ccf4100bb0b9c07baead16d. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250622 13:53:51.099834 5524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:51.100276 5524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:51.100390 5524 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250622 13:53:51.782775 5612 consensus_peers.cc:487] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 -> Peer b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899): Couldn't send request to peer b529cf0a8ccf4100bb0b9c07baead16d. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250622 13:53:53.100841 5524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:53.101413 5524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:53.101526 5524 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250622 13:53:53.554533 5612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39899: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250622 13:53:54.464557 5612 consensus_peers.cc:487] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 -> Peer b529cf0a8ccf4100bb0b9c07baead16d (127.0.103.65:39899): Couldn't send request to peer b529cf0a8ccf4100bb0b9c07baead16d. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250622 13:53:55.101876 5524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250622 13:53:55.102351 5524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250622 13:53:55.102430 5524 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250622 13:53:55.528143 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:53:55.545588 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:55.545835 413 tablet_replica.cc:331] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8: stopping tablet replica
I20250622 13:53:55.546026 413 raft_consensus.cc:2241] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:53:55.546380 413 raft_consensus.cc:2270] T 9ac76cf36fb84da78b209c977087daac P bf17379793784cd3a8862e6688948fc8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:55.564862 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:53:55.567873 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:53:55.577236 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:55.577486 413 tablet_replica.cc:331] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366: stopping tablet replica
I20250622 13:53:55.577659 413 raft_consensus.cc:2241] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:55.577839 413 raft_consensus.cc:2270] T 9ac76cf36fb84da78b209c977087daac P 6f75feff0d924a178373e475de012366 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:55.591162 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:53:55.594321 413 master.cc:561] Master@127.0.103.126:44135 shutting down...
I20250622 13:53:55.602764 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:53:55.602990 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:55.603081 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 376e98a8e8924773b022dbfb36da272a: stopping tablet replica
I20250622 13:53:55.616397 413 master.cc:583] Master@127.0.103.126:44135 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12589 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20250622 13:53:55.627336 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:39429
I20250622 13:53:55.627807 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:55.630544 5816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:55.630544 5817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:55.630692 5819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:55.630843 413 server_base.cc:1048] running on GCE node
I20250622 13:53:55.631340 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:55.631434 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:55.631479 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600435631465 us; error 0 us; skew 500 ppm
I20250622 13:53:55.631654 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:55.632874 413 webserver.cc:469] Webserver started at http://127.0.103.126:41797/ using document root <none> and password file <none>
I20250622 13:53:55.633080 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:55.633147 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:55.633255 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:55.633909 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "99d072fed1d94108a1d8c3219baf9ab9"
format_stamp: "Formatted at 2025-06-22 13:53:55 on dist-test-slave-tmv0"
I20250622 13:53:55.636260 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250622 13:53:55.637756 5824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.638074 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250622 13:53:55.638171 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "99d072fed1d94108a1d8c3219baf9ab9"
format_stamp: "Formatted at 2025-06-22 13:53:55 on dist-test-slave-tmv0"
I20250622 13:53:55.638299 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:55.654860 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:55.655418 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:55.665791 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39429
I20250622 13:53:55.665813 5875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:39429 every 8 connection(s)
I20250622 13:53:55.666972 5876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:55.670837 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9: Bootstrap starting.
I20250622 13:53:55.672102 5876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:55.673643 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9: No bootstrap required, opened a new log
I20250622 13:53:55.674216 5876 raft_consensus.cc:357] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d072fed1d94108a1d8c3219baf9ab9" member_type: VOTER }
I20250622 13:53:55.674337 5876 raft_consensus.cc:383] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:55.674436 5876 raft_consensus.cc:738] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d072fed1d94108a1d8c3219baf9ab9, State: Initialized, Role: FOLLOWER
I20250622 13:53:55.674659 5876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d072fed1d94108a1d8c3219baf9ab9" member_type: VOTER }
I20250622 13:53:55.674765 5876 raft_consensus.cc:397] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:53:55.674820 5876 raft_consensus.cc:491] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:53:55.674902 5876 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:55.676505 5876 raft_consensus.cc:513] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d072fed1d94108a1d8c3219baf9ab9" member_type: VOTER }
I20250622 13:53:55.676678 5876 leader_election.cc:304] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99d072fed1d94108a1d8c3219baf9ab9; no voters:
I20250622 13:53:55.677026 5876 leader_election.cc:290] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:53:55.677139 5879 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:55.677578 5879 raft_consensus.cc:695] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 99d072fed1d94108a1d8c3219baf9ab9, State: Running, Role: LEADER
I20250622 13:53:55.677835 5876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:53:55.677800 5879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d072fed1d94108a1d8c3219baf9ab9" member_type: VOTER }
I20250622 13:53:55.678589 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99d072fed1d94108a1d8c3219baf9ab9. Latest consensus state: current_term: 1 leader_uuid: "99d072fed1d94108a1d8c3219baf9ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d072fed1d94108a1d8c3219baf9ab9" member_type: VOTER } }
I20250622 13:53:55.678758 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [sys.catalog]: This master's current role is: LEADER
I20250622 13:53:55.679004 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99d072fed1d94108a1d8c3219baf9ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d072fed1d94108a1d8c3219baf9ab9" member_type: VOTER } }
I20250622 13:53:55.679209 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [sys.catalog]: This master's current role is: LEADER
I20250622 13:53:55.680183 5884 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:53:55.682181 5884 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:53:55.683149 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:53:55.685061 5884 catalog_manager.cc:1349] Generated new cluster ID: 06ab0d98334d4461bf0f5887cff4bf50
I20250622 13:53:55.685149 5884 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:53:55.692517 5884 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:53:55.693266 5884 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:53:55.702117 5884 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9: Generated new TSK 0
I20250622 13:53:55.702346 5884 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:53:55.716056 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:55.718907 5897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:55.719005 5898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:55.719619 5900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:55.720530 413 server_base.cc:1048] running on GCE node
I20250622 13:53:55.720791 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:55.720861 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:55.720906 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600435720891 us; error 0 us; skew 500 ppm
I20250622 13:53:55.721024 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:55.722301 413 webserver.cc:469] Webserver started at http://127.0.103.65:40181/ using document root <none> and password file <none>
I20250622 13:53:55.722538 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:55.722617 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:55.722728 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:55.723325 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "1588f0f54ac744a489c91d50b6f692fd"
format_stamp: "Formatted at 2025-06-22 13:53:55 on dist-test-slave-tmv0"
I20250622 13:53:55.725826 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250622 13:53:55.727461 5905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.727816 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:53:55.727948 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "1588f0f54ac744a489c91d50b6f692fd"
format_stamp: "Formatted at 2025-06-22 13:53:55 on dist-test-slave-tmv0"
I20250622 13:53:55.728065 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:55.768142 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:55.768726 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:55.769304 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:55.770215 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:55.770293 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.770401 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:55.770459 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.783767 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38627
I20250622 13:53:55.783797 5967 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38627 every 8 connection(s)
I20250622 13:53:55.785307 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:55.788129 5972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:55.788682 5973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:55.789568 5975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:55.789597 413 server_base.cc:1048] running on GCE node
I20250622 13:53:55.790007 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:55.790071 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:55.790132 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600435790112 us; error 0 us; skew 500 ppm
I20250622 13:53:55.790297 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:55.792169 413 webserver.cc:469] Webserver started at http://127.0.103.66:44679/ using document root <none> and password file <none>
I20250622 13:53:55.792446 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:55.792572 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:55.792726 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:55.793136 5968 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39429
I20250622 13:53:55.793277 5968 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:55.793457 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "5db8c0f5bd584a39a33146871659b264"
format_stamp: "Formatted at 2025-06-22 13:53:55 on dist-test-slave-tmv0"
I20250622 13:53:55.793666 5968 heartbeater.cc:507] Master 127.0.103.126:39429 requested a full tablet report, sending...
I20250622 13:53:55.794567 5841 ts_manager.cc:194] Registered new tserver with Master: 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627)
I20250622 13:53:55.796111 5841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58114
I20250622 13:53:55.796504 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250622 13:53:55.798159 5980 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.798473 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:53:55.798583 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "5db8c0f5bd584a39a33146871659b264"
format_stamp: "Formatted at 2025-06-22 13:53:55 on dist-test-slave-tmv0"
I20250622 13:53:55.798694 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:55.806957 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:55.807595 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:55.808177 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:55.809026 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:55.809106 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.809214 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:55.809278 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.821779 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:37955
I20250622 13:53:55.821799 6042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:37955 every 8 connection(s)
I20250622 13:53:55.823338 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:55.826794 6047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:55.827116 6046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:55.827281 6049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:55.828071 413 server_base.cc:1048] running on GCE node
I20250622 13:53:55.830893 6043 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39429
I20250622 13:53:55.831013 6043 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:55.831353 6043 heartbeater.cc:507] Master 127.0.103.126:39429 requested a full tablet report, sending...
I20250622 13:53:55.832154 5841 ts_manager.cc:194] Registered new tserver with Master: 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:55.832888 5841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58124
I20250622 13:53:55.842777 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:55.842873 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:55.842936 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600435842910 us; error 0 us; skew 500 ppm
I20250622 13:53:55.843078 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:55.844372 413 webserver.cc:469] Webserver started at http://127.0.103.67:46199/ using document root <none> and password file <none>
I20250622 13:53:55.844565 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:55.844633 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:55.844774 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:55.845343 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "6946c521f740456dbe01946108d1986b"
format_stamp: "Formatted at 2025-06-22 13:53:55 on dist-test-slave-tmv0"
I20250622 13:53:55.848455 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250622 13:53:55.850117 6054 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.850481 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:53:55.850580 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "6946c521f740456dbe01946108d1986b"
format_stamp: "Formatted at 2025-06-22 13:53:55 on dist-test-slave-tmv0"
I20250622 13:53:55.850713 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:55.868264 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:55.868808 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:55.869341 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:55.870141 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:55.870195 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.870270 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:55.870306 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:55.882016 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:35437
I20250622 13:53:55.882030 6116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:35437 every 8 connection(s)
I20250622 13:53:55.889012 6117 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39429
I20250622 13:53:55.889134 6117 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:55.889472 6117 heartbeater.cc:507] Master 127.0.103.126:39429 requested a full tablet report, sending...
I20250622 13:53:55.890254 5841 ts_manager.cc:194] Registered new tserver with Master: 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:53:55.891011 5841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58128
I20250622 13:53:55.891047 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008057336s
I20250622 13:53:56.797598 5968 heartbeater.cc:499] Master 127.0.103.126:39429 was elected leader, sending a full tablet report...
I20250622 13:53:56.834159 6043 heartbeater.cc:499] Master 127.0.103.126:39429 was elected leader, sending a full tablet report...
I20250622 13:53:56.892258 6117 heartbeater.cc:499] Master 127.0.103.126:39429 was elected leader, sending a full tablet report...
I20250622 13:53:56.918670 413 test_util.cc:276] Using random seed: 1831840110
I20250622 13:53:56.933902 5841 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58134:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:53:56.934505 5841 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:53:56.953713 5932 tablet_service.cc:1468] Processing CreateTablet for tablet c01ef28c08a24ed48fa6cbb8d7af60bd (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:53:56.953912 5930 tablet_service.cc:1468] Processing CreateTablet for tablet f1a0591bf6b349dabdb48e2474adf87d (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:53:56.954066 6082 tablet_service.cc:1468] Processing CreateTablet for tablet 774f8894e66b49e49d745aa79721654b (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:53:56.954232 5932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01ef28c08a24ed48fa6cbb8d7af60bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.954294 6081 tablet_service.cc:1468] Processing CreateTablet for tablet c01ef28c08a24ed48fa6cbb8d7af60bd (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:53:56.954490 6082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774f8894e66b49e49d745aa79721654b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.954473 5931 tablet_service.cc:1468] Processing CreateTablet for tablet 269be2c1778446f393fcca7c3314105e (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:53:56.954854 6080 tablet_service.cc:1468] Processing CreateTablet for tablet 269be2c1778446f393fcca7c3314105e (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:53:56.955221 6080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 269be2c1778446f393fcca7c3314105e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.955165 5933 tablet_service.cc:1468] Processing CreateTablet for tablet 774f8894e66b49e49d745aa79721654b (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:53:56.956014 6079 tablet_service.cc:1468] Processing CreateTablet for tablet f1a0591bf6b349dabdb48e2474adf87d (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:53:56.956372 6079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a0591bf6b349dabdb48e2474adf87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.961304 6137 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b: Bootstrap starting.
I20250622 13:53:56.961671 6081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01ef28c08a24ed48fa6cbb8d7af60bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.962937 6137 tablet_bootstrap.cc:654] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:56.963971 6138 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd: Bootstrap starting.
I20250622 13:53:56.964941 6137 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b: No bootstrap required, opened a new log
I20250622 13:53:56.965062 6137 ts_tablet_manager.cc:1397] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250622 13:53:56.954861 5931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 269be2c1778446f393fcca7c3314105e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.965903 6137 raft_consensus.cc:357] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:56.966089 6137 raft_consensus.cc:383] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:56.966151 6137 raft_consensus.cc:738] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6946c521f740456dbe01946108d1986b, State: Initialized, Role: FOLLOWER
I20250622 13:53:56.966429 6137 consensus_queue.cc:260] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:56.967245 6137 ts_tablet_manager.cc:1428] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:56.967494 6138 tablet_bootstrap.cc:654] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:56.967856 6137 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b: Bootstrap starting.
I20250622 13:53:56.969516 5930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a0591bf6b349dabdb48e2474adf87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.969522 6137 tablet_bootstrap.cc:654] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:56.973536 5933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774f8894e66b49e49d745aa79721654b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.975986 6137 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b: No bootstrap required, opened a new log
I20250622 13:53:56.975986 6138 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd: No bootstrap required, opened a new log
I20250622 13:53:56.976171 6137 ts_tablet_manager.cc:1397] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20250622 13:53:56.976171 6138 ts_tablet_manager.cc:1397] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20250622 13:53:56.976990 6138 raft_consensus.cc:357] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:56.976990 6137 raft_consensus.cc:357] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:56.977334 6138 raft_consensus.cc:383] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:56.977385 6137 raft_consensus.cc:383] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:56.977430 6138 raft_consensus.cc:738] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1588f0f54ac744a489c91d50b6f692fd, State: Initialized, Role: FOLLOWER
I20250622 13:53:56.977437 6137 raft_consensus.cc:738] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6946c521f740456dbe01946108d1986b, State: Initialized, Role: FOLLOWER
I20250622 13:53:56.977697 6137 consensus_queue.cc:260] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:56.977697 6138 consensus_queue.cc:260] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:56.978302 6137 ts_tablet_manager.cc:1428] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:53:56.980096 6138 ts_tablet_manager.cc:1428] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250622 13:53:56.985532 6138 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd: Bootstrap starting.
I20250622 13:53:56.986506 6137 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b: Bootstrap starting.
I20250622 13:53:56.987016 6138 tablet_bootstrap.cc:654] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:56.987107 6007 tablet_service.cc:1468] Processing CreateTablet for tablet c01ef28c08a24ed48fa6cbb8d7af60bd (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:53:56.987306 6008 tablet_service.cc:1468] Processing CreateTablet for tablet 774f8894e66b49e49d745aa79721654b (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:53:56.987610 6007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01ef28c08a24ed48fa6cbb8d7af60bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.988166 6137 tablet_bootstrap.cc:654] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:56.988037 6006 tablet_service.cc:1468] Processing CreateTablet for tablet 269be2c1778446f393fcca7c3314105e (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:53:56.988556 6006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 269be2c1778446f393fcca7c3314105e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.992444 6005 tablet_service.cc:1468] Processing CreateTablet for tablet f1a0591bf6b349dabdb48e2474adf87d (DEFAULT_TABLE table=test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:53:56.992914 6005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a0591bf6b349dabdb48e2474adf87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.997314 6008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774f8894e66b49e49d745aa79721654b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:56.997720 6146 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264: Bootstrap starting.
I20250622 13:53:57.000293 6146 tablet_bootstrap.cc:654] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:57.010128 6146 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264: No bootstrap required, opened a new log
I20250622 13:53:57.010381 6146 ts_tablet_manager.cc:1397] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20250622 13:53:57.011510 6146 raft_consensus.cc:357] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.011858 6146 raft_consensus.cc:383] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.011981 6146 raft_consensus.cc:738] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db8c0f5bd584a39a33146871659b264, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.012337 6146 consensus_queue.cc:260] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.017132 6146 ts_tablet_manager.cc:1428] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250622 13:53:57.019598 6146 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264: Bootstrap starting.
I20250622 13:53:57.022029 6146 tablet_bootstrap.cc:654] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:57.024912 6137 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b: No bootstrap required, opened a new log
I20250622 13:53:57.024912 6138 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd: No bootstrap required, opened a new log
I20250622 13:53:57.025043 6146 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264: No bootstrap required, opened a new log
I20250622 13:53:57.025071 6137 ts_tablet_manager.cc:1397] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.031s
I20250622 13:53:57.025094 6138 ts_tablet_manager.cc:1397] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.027s
I20250622 13:53:57.025130 6146 ts_tablet_manager.cc:1397] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250622 13:53:57.025748 6137 raft_consensus.cc:357] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.025892 6137 raft_consensus.cc:383] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.025887 6146 raft_consensus.cc:357] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.026037 6137 raft_consensus.cc:738] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6946c521f740456dbe01946108d1986b, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.026057 6146 raft_consensus.cc:383] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.025969 6138 raft_consensus.cc:357] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.026151 6138 raft_consensus.cc:383] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.026127 6146 raft_consensus.cc:738] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db8c0f5bd584a39a33146871659b264, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.026211 6138 raft_consensus.cc:738] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1588f0f54ac744a489c91d50b6f692fd, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.026254 6137 consensus_queue.cc:260] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.026402 6146 consensus_queue.cc:260] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.026420 6138 consensus_queue.cc:260] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.026876 6146 ts_tablet_manager.cc:1428] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:53:57.026897 6138 ts_tablet_manager.cc:1428] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250622 13:53:57.027117 6146 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264: Bootstrap starting.
I20250622 13:53:57.027276 6137 ts_tablet_manager.cc:1428] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:53:57.027491 6137 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b: Bootstrap starting.
I20250622 13:53:57.028167 6138 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd: Bootstrap starting.
I20250622 13:53:57.028996 6146 tablet_bootstrap.cc:654] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:57.029532 6137 tablet_bootstrap.cc:654] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:57.030143 6138 tablet_bootstrap.cc:654] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:57.038185 6146 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264: No bootstrap required, opened a new log
I20250622 13:53:57.038314 6146 ts_tablet_manager.cc:1397] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20250622 13:53:57.038470 6137 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b: No bootstrap required, opened a new log
I20250622 13:53:57.038595 6137 ts_tablet_manager.cc:1397] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20250622 13:53:57.038885 6138 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd: No bootstrap required, opened a new log
I20250622 13:53:57.038995 6138 ts_tablet_manager.cc:1397] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20250622 13:53:57.039084 6146 raft_consensus.cc:357] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.039258 6146 raft_consensus.cc:383] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.039258 6137 raft_consensus.cc:357] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.039374 6146 raft_consensus.cc:738] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db8c0f5bd584a39a33146871659b264, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.039445 6137 raft_consensus.cc:383] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.039507 6137 raft_consensus.cc:738] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6946c521f740456dbe01946108d1986b, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.039636 6138 raft_consensus.cc:357] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:57.039693 6137 consensus_queue.cc:260] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.039826 6138 raft_consensus.cc:383] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.039636 6146 consensus_queue.cc:260] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.039883 6138 raft_consensus.cc:738] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1588f0f54ac744a489c91d50b6f692fd, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.040228 6146 ts_tablet_manager.cc:1428] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:53:57.040222 6138 consensus_queue.cc:260] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:57.040499 6146 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264: Bootstrap starting.
I20250622 13:53:57.040647 6138 ts_tablet_manager.cc:1428] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250622 13:53:57.040845 6138 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd: Bootstrap starting.
I20250622 13:53:57.042271 6146 tablet_bootstrap.cc:654] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:57.042462 6138 tablet_bootstrap.cc:654] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd: Neither blocks nor log segments found. Creating new log.
I20250622 13:53:57.044270 6137 ts_tablet_manager.cc:1428] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250622 13:53:57.047245 6138 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd: No bootstrap required, opened a new log
I20250622 13:53:57.047339 6138 ts_tablet_manager.cc:1397] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250622 13:53:57.047377 6146 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264: No bootstrap required, opened a new log
I20250622 13:53:57.047495 6146 ts_tablet_manager.cc:1397] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250622 13:53:57.047871 6138 raft_consensus.cc:357] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.047982 6138 raft_consensus.cc:383] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.048012 6138 raft_consensus.cc:738] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1588f0f54ac744a489c91d50b6f692fd, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.047993 6146 raft_consensus.cc:357] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:57.048172 6146 raft_consensus.cc:383] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:53:57.048231 6146 raft_consensus.cc:738] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db8c0f5bd584a39a33146871659b264, State: Initialized, Role: FOLLOWER
I20250622 13:53:57.048213 6138 consensus_queue.cc:260] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.048419 6146 consensus_queue.cc:260] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:57.048673 6138 ts_tablet_manager.cc:1428] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250622 13:53:57.048833 6146 ts_tablet_manager.cc:1428] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:53:57.065444 6143 raft_consensus.cc:491] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:57.065580 6143 raft_consensus.cc:513] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.066227 6143 leader_election.cc:290] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955), 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:53:57.066901 6143 raft_consensus.cc:491] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:57.067027 6143 raft_consensus.cc:513] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.067602 6143 leader_election.cc:290] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955), 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:53:57.072988 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a0591bf6b349dabdb48e2474adf87d" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264" is_pre_election: true
I20250622 13:53:57.073037 6017 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c01ef28c08a24ed48fa6cbb8d7af60bd" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264" is_pre_election: true
I20250622 13:53:57.072990 6092 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a0591bf6b349dabdb48e2474adf87d" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6946c521f740456dbe01946108d1986b" is_pre_election: true
I20250622 13:53:57.073062 6091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c01ef28c08a24ed48fa6cbb8d7af60bd" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6946c521f740456dbe01946108d1986b" is_pre_election: true
I20250622 13:53:57.073217 6018 raft_consensus.cc:2466] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 0.
I20250622 13:53:57.073271 6091 raft_consensus.cc:2466] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 0.
I20250622 13:53:57.073256 6017 raft_consensus.cc:2466] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 0.
I20250622 13:53:57.073256 6092 raft_consensus.cc:2466] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 0.
I20250622 13:53:57.073630 5909 leader_election.cc:304] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1588f0f54ac744a489c91d50b6f692fd, 5db8c0f5bd584a39a33146871659b264; no voters:
I20250622 13:53:57.073621 5907 leader_election.cc:304] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1588f0f54ac744a489c91d50b6f692fd, 6946c521f740456dbe01946108d1986b; no voters:
I20250622 13:53:57.073918 6143 raft_consensus.cc:2802] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:57.073976 6143 raft_consensus.cc:491] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:57.074056 6143 raft_consensus.cc:3058] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.074355 6157 raft_consensus.cc:2802] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:57.074421 6157 raft_consensus.cc:491] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:57.074498 6157 raft_consensus.cc:3058] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.075968 6143 raft_consensus.cc:513] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.076102 6157 raft_consensus.cc:513] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.076499 6143 leader_election.cc:290] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 election: Requested vote from peers 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955), 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:53:57.076646 6157 leader_election.cc:290] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 election: Requested vote from peers 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955), 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:53:57.076946 6017 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a0591bf6b349dabdb48e2474adf87d" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264"
I20250622 13:53:57.077143 6017 raft_consensus.cc:3058] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.077137 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c01ef28c08a24ed48fa6cbb8d7af60bd" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264"
I20250622 13:53:57.077286 6092 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a0591bf6b349dabdb48e2474adf87d" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6946c521f740456dbe01946108d1986b"
I20250622 13:53:57.077334 6018 raft_consensus.cc:3058] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.077418 6091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c01ef28c08a24ed48fa6cbb8d7af60bd" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6946c521f740456dbe01946108d1986b"
I20250622 13:53:57.077461 6092 raft_consensus.cc:3058] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.077559 6091 raft_consensus.cc:3058] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.080260 6017 raft_consensus.cc:2466] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 1.
I20250622 13:53:57.080380 6018 raft_consensus.cc:2466] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 1.
I20250622 13:53:57.080381 6091 raft_consensus.cc:2466] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 1.
I20250622 13:53:57.080261 6092 raft_consensus.cc:2466] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 1.
I20250622 13:53:57.080701 5907 leader_election.cc:304] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1588f0f54ac744a489c91d50b6f692fd, 6946c521f740456dbe01946108d1986b; no voters:
I20250622 13:53:57.080749 5909 leader_election.cc:304] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1588f0f54ac744a489c91d50b6f692fd, 5db8c0f5bd584a39a33146871659b264; no voters:
I20250622 13:53:57.080963 6157 raft_consensus.cc:2802] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:57.080945 6143 raft_consensus.cc:2802] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:57.081454 6157 raft_consensus.cc:695] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: Becoming Leader. State: Replica: 1588f0f54ac744a489c91d50b6f692fd, State: Running, Role: LEADER
I20250622 13:53:57.081506 6143 raft_consensus.cc:695] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: Becoming Leader. State: Replica: 1588f0f54ac744a489c91d50b6f692fd, State: Running, Role: LEADER
I20250622 13:53:57.081722 6157 consensus_queue.cc:237] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.081775 6143 consensus_queue.cc:237] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.084192 5841 catalog_manager.cc:5582] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65). New cstate: current_term: 1 leader_uuid: "1588f0f54ac744a489c91d50b6f692fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:57.087543 5841 catalog_manager.cc:5582] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65). New cstate: current_term: 1 leader_uuid: "1588f0f54ac744a489c91d50b6f692fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:57.219547 6140 raft_consensus.cc:491] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:57.220183 6140 raft_consensus.cc:513] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.221323 6140 leader_election.cc:290] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:57.227331 5943 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "269be2c1778446f393fcca7c3314105e" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1588f0f54ac744a489c91d50b6f692fd" is_pre_election: true
I20250622 13:53:57.227567 5943 raft_consensus.cc:2466] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 0.
I20250622 13:53:57.228003 6057 leader_election.cc:304] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1588f0f54ac744a489c91d50b6f692fd, 6946c521f740456dbe01946108d1986b; no voters:
I20250622 13:53:57.228276 6140 raft_consensus.cc:2802] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:57.228372 6140 raft_consensus.cc:491] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:57.228478 6140 raft_consensus.cc:3058] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.230351 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "269be2c1778446f393fcca7c3314105e" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264" is_pre_election: true
I20250622 13:53:57.230547 6018 raft_consensus.cc:2466] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 0.
I20250622 13:53:57.230898 6140 raft_consensus.cc:513] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.231458 6140 leader_election.cc:290] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 1 election: Requested vote from peers 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:57.231765 5943 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "269be2c1778446f393fcca7c3314105e" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1588f0f54ac744a489c91d50b6f692fd"
I20250622 13:53:57.231947 5943 raft_consensus.cc:3058] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.231946 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "269be2c1778446f393fcca7c3314105e" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264"
I20250622 13:53:57.232137 6018 raft_consensus.cc:3058] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.234639 5943 raft_consensus.cc:2466] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 1.
I20250622 13:53:57.234639 6018 raft_consensus.cc:2466] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 1.
I20250622 13:53:57.235116 6058 leader_election.cc:304] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1588f0f54ac744a489c91d50b6f692fd, 5db8c0f5bd584a39a33146871659b264, 6946c521f740456dbe01946108d1986b; no voters:
I20250622 13:53:57.235445 6140 raft_consensus.cc:2802] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:57.235934 6140 raft_consensus.cc:695] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 1 LEADER]: Becoming Leader. State: Replica: 6946c521f740456dbe01946108d1986b, State: Running, Role: LEADER
I20250622 13:53:57.236263 6140 consensus_queue.cc:237] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:57.239706 5841 catalog_manager.cc:5582] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6946c521f740456dbe01946108d1986b (127.0.103.67). New cstate: current_term: 1 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } }
I20250622 13:53:57.289722 6157 raft_consensus.cc:491] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:57.289971 6157 raft_consensus.cc:513] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:57.290850 6157 leader_election.cc:290] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6946c521f740456dbe01946108d1986b (127.0.103.67:35437), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:57.291353 6092 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "774f8894e66b49e49d745aa79721654b" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6946c521f740456dbe01946108d1986b" is_pre_election: true
I20250622 13:53:57.291388 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "774f8894e66b49e49d745aa79721654b" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264" is_pre_election: true
I20250622 13:53:57.291558 6092 raft_consensus.cc:2466] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 0.
I20250622 13:53:57.291595 6018 raft_consensus.cc:2466] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 0.
I20250622 13:53:57.292069 5907 leader_election.cc:304] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1588f0f54ac744a489c91d50b6f692fd, 6946c521f740456dbe01946108d1986b; no voters:
I20250622 13:53:57.292397 6157 raft_consensus.cc:2802] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:53:57.292497 6157 raft_consensus.cc:491] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:57.292596 6157 raft_consensus.cc:3058] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.294531 6157 raft_consensus.cc:513] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:57.295086 6157 leader_election.cc:290] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 election: Requested vote from peers 6946c521f740456dbe01946108d1986b (127.0.103.67:35437), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:57.295563 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "774f8894e66b49e49d745aa79721654b" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264"
I20250622 13:53:57.295572 6092 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "774f8894e66b49e49d745aa79721654b" candidate_uuid: "1588f0f54ac744a489c91d50b6f692fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6946c521f740456dbe01946108d1986b"
I20250622 13:53:57.295732 6018 raft_consensus.cc:3058] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.295835 6092 raft_consensus.cc:3058] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:53:57.297652 6018 raft_consensus.cc:2466] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 1.
I20250622 13:53:57.298007 5909 leader_election.cc:304] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1588f0f54ac744a489c91d50b6f692fd, 5db8c0f5bd584a39a33146871659b264; no voters:
I20250622 13:53:57.298246 6157 raft_consensus.cc:2802] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:53:57.298409 6157 raft_consensus.cc:695] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: Becoming Leader. State: Replica: 1588f0f54ac744a489c91d50b6f692fd, State: Running, Role: LEADER
I20250622 13:53:57.298892 6157 consensus_queue.cc:237] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:57.299132 6092 raft_consensus.cc:2466] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1588f0f54ac744a489c91d50b6f692fd in term 1.
I20250622 13:53:57.301501 5841 catalog_manager.cc:5582] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65). New cstate: current_term: 1 leader_uuid: "1588f0f54ac744a489c91d50b6f692fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:57.319898 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:53:57.343621 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:53:57.344092 413 tablet_replica.cc:331] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd: stopping tablet replica
I20250622 13:53:57.344393 413 raft_consensus.cc:2241] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:53:57.344621 413 raft_consensus.cc:2270] T 269be2c1778446f393fcca7c3314105e P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:57.345559 413 tablet_replica.cc:331] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd: stopping tablet replica
I20250622 13:53:57.345774 413 raft_consensus.cc:2241] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:53:57.346079 413 pending_rounds.cc:70] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd: Trying to abort 1 pending ops.
I20250622 13:53:57.346151 413 pending_rounds.cc:77] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459390289178624 op_type: NO_OP noop_request { }
I20250622 13:53:57.346304 413 raft_consensus.cc:2887] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:53:57.346444 413 raft_consensus.cc:2270] T c01ef28c08a24ed48fa6cbb8d7af60bd P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:57.347322 413 tablet_replica.cc:331] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd: stopping tablet replica
I20250622 13:53:57.347514 413 raft_consensus.cc:2241] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:53:57.347767 413 pending_rounds.cc:70] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd: Trying to abort 1 pending ops.
I20250622 13:53:57.347841 413 pending_rounds.cc:77] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459391178747904 op_type: NO_OP noop_request { }
I20250622 13:53:57.347963 413 raft_consensus.cc:2887] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:53:57.348085 413 raft_consensus.cc:2270] T 774f8894e66b49e49d745aa79721654b P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:57.349085 413 tablet_replica.cc:331] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd: stopping tablet replica
I20250622 13:53:57.349273 413 raft_consensus.cc:2241] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:53:57.349570 413 pending_rounds.cc:70] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd: Trying to abort 1 pending ops.
I20250622 13:53:57.349658 413 pending_rounds.cc:77] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459390289256448 op_type: NO_OP noop_request { }
I20250622 13:53:57.349772 413 raft_consensus.cc:2887] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:53:57.349908 413 raft_consensus.cc:2270] T f1a0591bf6b349dabdb48e2474adf87d P 1588f0f54ac744a489c91d50b6f692fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:53:57.608379 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:53:57.717844 6140 consensus_queue.cc:1035] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:53:57.724004 6057 consensus_peers.cc:487] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:58.285887 6170 consensus_queue.cc:579] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Leader has been unable to successfully communicate with peer 1588f0f54ac744a489c91d50b6f692fd for more than 1 seconds (1.049s)
W20250622 13:53:58.288889 5840 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 269be2c1778446f393fcca7c3314105e with cas_config_opid_index -1: no extra replica candidate found for tablet 269be2c1778446f393fcca7c3314105e (table test-workload [id=7b8847bbd85a49eb93bf4d90868d7fc6]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250622 13:53:58.612856 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:53:58.615628 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:53:58.615760 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:58.616611 413 server_base.cc:1048] running on GCE node
W20250622 13:53:58.616837 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:53:58.619042 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:53:58.619122 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:53:58.619164 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600438619144 us; error 0 us; skew 500 ppm
I20250622 13:53:58.619319 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:53:58.626049 413 webserver.cc:469] Webserver started at http://127.0.103.68:40967/ using document root <none> and password file <none>
I20250622 13:53:58.626304 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:53:58.626379 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:53:58.626495 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:53:58.627171 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-3-root/instance:
uuid: "ac521567b5ce4e179be2551ef1363f84"
format_stamp: "Formatted at 2025-06-22 13:53:58 on dist-test-slave-tmv0"
I20250622 13:53:58.629801 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:53:58.631457 6185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:58.631927 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:53:58.632045 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-3-root
uuid: "ac521567b5ce4e179be2551ef1363f84"
format_stamp: "Formatted at 2025-06-22 13:53:58 on dist-test-slave-tmv0"
I20250622 13:53:58.632155 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750600281110842-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:53:58.639724 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:53:58.640249 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:53:58.640774 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:53:58.641654 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:53:58.641719 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:58.641793 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:53:58.641837 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:53:58.654197 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:38175
I20250622 13:53:58.654220 6247 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:38175 every 8 connection(s)
I20250622 13:53:58.661051 6248 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39429
I20250622 13:53:58.661207 6248 heartbeater.cc:461] Registering TS with master...
I20250622 13:53:58.661659 6248 heartbeater.cc:507] Master 127.0.103.126:39429 requested a full tablet report, sending...
I20250622 13:53:58.662493 5840 ts_manager.cc:194] Registered new tserver with Master: ac521567b5ce4e179be2551ef1363f84 (127.0.103.68:38175)
I20250622 13:53:58.663296 5840 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58138
W20250622 13:53:58.684077 5889 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1588f0f54ac744a489c91d50b6f692fd not available for placement
I20250622 13:53:58.801045 6174 raft_consensus.cc:491] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:58.801249 6174 raft_consensus.cc:513] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:58.801946 6174 leader_election.cc:290] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:58.802678 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a0591bf6b349dabdb48e2474adf87d" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264" is_pre_election: true
W20250622 13:53:58.802857 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20250622 13:53:58.802871 6018 raft_consensus.cc:2466] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 1.
I20250622 13:53:58.803435 6058 leader_election.cc:304] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db8c0f5bd584a39a33146871659b264, 6946c521f740456dbe01946108d1986b; no voters:
I20250622 13:53:58.803697 6174 raft_consensus.cc:2802] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:53:58.803766 6174 raft_consensus.cc:491] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:58.803829 6174 raft_consensus.cc:3058] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Advancing to term 2
W20250622 13:53:58.804122 6057 leader_election.cc:336] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:53:58.805959 6174 raft_consensus.cc:513] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:58.806478 6174 leader_election.cc:290] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: Requested vote from peers 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:58.807003 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1a0591bf6b349dabdb48e2474adf87d" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264"
I20250622 13:53:58.807190 6018 raft_consensus.cc:3058] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Advancing to term 2
W20250622 13:53:58.807890 6057 leader_election.cc:336] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:53:58.809481 6018 raft_consensus.cc:2466] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 2.
I20250622 13:53:58.809834 6058 leader_election.cc:304] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5db8c0f5bd584a39a33146871659b264, 6946c521f740456dbe01946108d1986b; no voters: 1588f0f54ac744a489c91d50b6f692fd
I20250622 13:53:58.810107 6174 raft_consensus.cc:2802] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:58.810431 6174 raft_consensus.cc:695] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Becoming Leader. State: Replica: 6946c521f740456dbe01946108d1986b, State: Running, Role: LEADER
I20250622 13:53:58.810667 6174 consensus_queue.cc:237] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:58.812801 5840 catalog_manager.cc:5582] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b reported cstate change: term changed from 1 to 2, leader changed from 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) to 6946c521f740456dbe01946108d1986b (127.0.103.67). New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } }
I20250622 13:53:58.852653 6174 raft_consensus.cc:491] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:58.852840 6174 raft_consensus.cc:513] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:58.853436 6174 leader_election.cc:290] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:58.853977 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "774f8894e66b49e49d745aa79721654b" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264" is_pre_election: true
I20250622 13:53:58.854183 6018 raft_consensus.cc:2466] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 1.
I20250622 13:53:58.854521 6058 leader_election.cc:304] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db8c0f5bd584a39a33146871659b264, 6946c521f740456dbe01946108d1986b; no voters:
I20250622 13:53:58.854744 6174 raft_consensus.cc:2802] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:53:58.854830 6174 raft_consensus.cc:491] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250622 13:53:58.854851 6057 leader_election.cc:336] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:53:58.854908 6174 raft_consensus.cc:3058] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:58.856768 6174 raft_consensus.cc:513] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:58.857306 6174 leader_election.cc:290] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: Requested vote from peers 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:58.857770 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "774f8894e66b49e49d745aa79721654b" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264"
I20250622 13:53:58.857923 6018 raft_consensus.cc:3058] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Advancing to term 2
W20250622 13:53:58.858400 6057 leader_election.cc:336] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:53:58.860191 6018 raft_consensus.cc:2466] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 2.
I20250622 13:53:58.860541 6058 leader_election.cc:304] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5db8c0f5bd584a39a33146871659b264, 6946c521f740456dbe01946108d1986b; no voters: 1588f0f54ac744a489c91d50b6f692fd
I20250622 13:53:58.860781 6174 raft_consensus.cc:2802] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:58.860886 6174 raft_consensus.cc:695] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Becoming Leader. State: Replica: 6946c521f740456dbe01946108d1986b, State: Running, Role: LEADER
I20250622 13:53:58.861106 6174 consensus_queue.cc:237] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } }
I20250622 13:53:58.863442 5840 catalog_manager.cc:5582] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b reported cstate change: term changed from 1 to 2, leader changed from 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) to 6946c521f740456dbe01946108d1986b (127.0.103.67). New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:58.877884 6174 raft_consensus.cc:491] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:53:58.878098 6174 raft_consensus.cc:513] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:58.878765 6174 leader_election.cc:290] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:58.879390 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c01ef28c08a24ed48fa6cbb8d7af60bd" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264" is_pre_election: true
I20250622 13:53:58.879586 6018 raft_consensus.cc:2466] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 1.
I20250622 13:53:58.879967 6058 leader_election.cc:304] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db8c0f5bd584a39a33146871659b264, 6946c521f740456dbe01946108d1986b; no voters:
I20250622 13:53:58.880290 6174 raft_consensus.cc:2802] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250622 13:53:58.880353 6057 leader_election.cc:336] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:53:58.880379 6174 raft_consensus.cc:491] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:53:58.880502 6174 raft_consensus.cc:3058] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:53:58.882517 6174 raft_consensus.cc:513] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:58.882993 6174 leader_election.cc:290] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: Requested vote from peers 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627), 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955)
I20250622 13:53:58.883500 6018 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c01ef28c08a24ed48fa6cbb8d7af60bd" candidate_uuid: "6946c521f740456dbe01946108d1986b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db8c0f5bd584a39a33146871659b264"
I20250622 13:53:58.883642 6018 raft_consensus.cc:3058] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Advancing to term 2
W20250622 13:53:58.884773 6057 leader_election.cc:336] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:53:58.885447 6018 raft_consensus.cc:2466] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6946c521f740456dbe01946108d1986b in term 2.
I20250622 13:53:58.885802 6058 leader_election.cc:304] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5db8c0f5bd584a39a33146871659b264, 6946c521f740456dbe01946108d1986b; no voters: 1588f0f54ac744a489c91d50b6f692fd
I20250622 13:53:58.886008 6174 raft_consensus.cc:2802] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:53:58.886132 6174 raft_consensus.cc:695] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Becoming Leader. State: Replica: 6946c521f740456dbe01946108d1986b, State: Running, Role: LEADER
I20250622 13:53:58.886359 6174 consensus_queue.cc:237] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } }
I20250622 13:53:58.888481 5840 catalog_manager.cc:5582] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b reported cstate change: term changed from 1 to 2, leader changed from 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) to 6946c521f740456dbe01946108d1986b (127.0.103.67). New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } }
I20250622 13:53:58.904359 6092 consensus_queue.cc:237] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:53:58.907317 6018 raft_consensus.cc:1273] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:53:58.907963 6174 consensus_queue.cc:1035] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250622 13:53:58.908188 6057 consensus_peers.cc:487] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:58.910638 6254 raft_consensus.cc:2953] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
I20250622 13:53:58.911475 6018 raft_consensus.cc:2953] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
W20250622 13:53:58.913977 6055 consensus_peers.cc:487] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b -> Peer ac521567b5ce4e179be2551ef1363f84 (127.0.103.68:38175): Couldn't send request to peer ac521567b5ce4e179be2551ef1363f84. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 269be2c1778446f393fcca7c3314105e. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:58.914506 5826 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 269be2c1778446f393fcca7c3314105e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250622 13:53:58.915966 5840 catalog_manager.cc:5582] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b reported cstate change: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250622 13:53:59.240757 6057 consensus_peers.cc:487] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:59.359138 6018 raft_consensus.cc:1273] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250622 13:53:59.359730 6254 consensus_queue.cc:1035] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:53:59.364298 6057 consensus_peers.cc:487] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:59.368687 6018 raft_consensus.cc:1273] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250622 13:53:59.369102 6263 consensus_queue.cc:1035] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:53:59.373440 6057 consensus_peers.cc:487] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:59.373803 6018 raft_consensus.cc:1273] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250622 13:53:59.374235 6174 consensus_queue.cc:1035] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:53:59.446281 6273 ts_tablet_manager.cc:927] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: Initiating tablet copy from peer 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:53:59.446836 6273 tablet_copy_client.cc:323] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35437
I20250622 13:53:59.452399 6102 tablet_copy_service.cc:140] P 6946c521f740456dbe01946108d1986b: Received BeginTabletCopySession request for tablet 269be2c1778446f393fcca7c3314105e from peer ac521567b5ce4e179be2551ef1363f84 ({username='slave'} at 127.0.0.1:52328)
I20250622 13:53:59.452548 6102 tablet_copy_service.cc:161] P 6946c521f740456dbe01946108d1986b: Beginning new tablet copy session on tablet 269be2c1778446f393fcca7c3314105e from peer ac521567b5ce4e179be2551ef1363f84 at {username='slave'} at 127.0.0.1:52328: session id = ac521567b5ce4e179be2551ef1363f84-269be2c1778446f393fcca7c3314105e
I20250622 13:53:59.454437 6102 tablet_copy_source_session.cc:215] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:53:59.455513 6273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 269be2c1778446f393fcca7c3314105e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:53:59.459520 6273 tablet_copy_client.cc:806] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: tablet copy: Starting download of 0 data blocks...
I20250622 13:53:59.459712 6273 tablet_copy_client.cc:670] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: tablet copy: Starting download of 1 WAL segments...
I20250622 13:53:59.461066 6273 tablet_copy_client.cc:538] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:53:59.463241 6273 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: Bootstrap starting.
I20250622 13:53:59.467823 6273 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:53:59.468135 6273 tablet_bootstrap.cc:492] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: Bootstrap complete.
I20250622 13:53:59.468336 6273 ts_tablet_manager.cc:1397] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250622 13:53:59.468958 6273 raft_consensus.cc:357] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:53:59.469102 6273 raft_consensus.cc:738] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ac521567b5ce4e179be2551ef1363f84, State: Initialized, Role: LEARNER
I20250622 13:53:59.469240 6273 consensus_queue.cc:260] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:53:59.469877 6273 ts_tablet_manager.cc:1428] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250622 13:53:59.469934 6248 heartbeater.cc:499] Master 127.0.103.126:39429 was elected leader, sending a full tablet report...
I20250622 13:53:59.470690 6102 tablet_copy_service.cc:342] P 6946c521f740456dbe01946108d1986b: Request end of tablet copy session ac521567b5ce4e179be2551ef1363f84-269be2c1778446f393fcca7c3314105e received from {username='slave'} at 127.0.0.1:52328
I20250622 13:53:59.470808 6102 tablet_copy_service.cc:434] P 6946c521f740456dbe01946108d1986b: ending tablet copy session ac521567b5ce4e179be2551ef1363f84-269be2c1778446f393fcca7c3314105e on tablet 269be2c1778446f393fcca7c3314105e with peer ac521567b5ce4e179be2551ef1363f84
W20250622 13:53:59.684865 5889 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1588f0f54ac744a489c91d50b6f692fd not available for placement
I20250622 13:53:59.849902 6223 raft_consensus.cc:1215] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250622 13:53:59.929695 6174 consensus_queue.cc:579] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Leader has been unable to successfully communicate with peer 1588f0f54ac744a489c91d50b6f692fd for more than 1 seconds (1.119s)
I20250622 13:53:59.930657 6174 consensus_queue.cc:579] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Leader has been unable to successfully communicate with peer 1588f0f54ac744a489c91d50b6f692fd for more than 1 seconds (1.044s)
I20250622 13:53:59.933611 6092 consensus_queue.cc:237] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
W20250622 13:53:59.935889 6055 consensus_peers.cc:487] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b -> Peer ac521567b5ce4e179be2551ef1363f84 (127.0.103.68:38175): Couldn't send request to peer ac521567b5ce4e179be2551ef1363f84. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1a0591bf6b349dabdb48e2474adf87d. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:59.936020 6018 raft_consensus.cc:1273] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:53:59.936137 6092 consensus_queue.cc:237] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:53:59.936429 6174 consensus_queue.cc:1035] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250622 13:53:59.938047 6055 consensus_peers.cc:487] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b -> Peer ac521567b5ce4e179be2551ef1363f84 (127.0.103.68:38175): Couldn't send request to peer ac521567b5ce4e179be2551ef1363f84. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c01ef28c08a24ed48fa6cbb8d7af60bd. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:59.938086 6017 raft_consensus.cc:1273] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250622 13:53:59.938215 6057 consensus_peers.cc:487] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:59.938493 6263 consensus_queue.cc:1035] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:53:59.938578 6174 raft_consensus.cc:2953] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
W20250622 13:53:59.939085 6057 consensus_peers.cc:487] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:53:59.941123 5826 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f1a0591bf6b349dabdb48e2474adf87d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 13:53:59.941533 6017 raft_consensus.cc:2953] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
I20250622 13:53:59.942656 6254 raft_consensus.cc:2953] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
I20250622 13:53:59.943274 6018 raft_consensus.cc:2953] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
I20250622 13:53:59.946017 5840 catalog_manager.cc:5582] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b reported cstate change: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:53:59.947077 5826 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c01ef28c08a24ed48fa6cbb8d7af60bd with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 13:53:59.955906 5841 catalog_manager.cc:5582] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 reported cstate change: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
I20250622 13:53:59.989496 6280 consensus_queue.cc:579] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Leader has been unable to successfully communicate with peer 1588f0f54ac744a489c91d50b6f692fd for more than 1 seconds (1.085s)
I20250622 13:54:00.327095 6280 consensus_queue.cc:579] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Leader has been unable to successfully communicate with peer 1588f0f54ac744a489c91d50b6f692fd for more than 1 seconds (1.466s)
I20250622 13:54:00.330735 6092 consensus_queue.cc:237] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:54:00.332502 6017 raft_consensus.cc:1273] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250622 13:54:00.332968 6055 consensus_peers.cc:487] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b -> Peer ac521567b5ce4e179be2551ef1363f84 (127.0.103.68:38175): Couldn't send request to peer ac521567b5ce4e179be2551ef1363f84. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 774f8894e66b49e49d745aa79721654b. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:00.332975 6280 consensus_queue.cc:1035] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250622 13:54:00.333215 6057 consensus_peers.cc:487] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:00.335045 6254 raft_consensus.cc:2953] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
I20250622 13:54:00.335667 6017 raft_consensus.cc:2953] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
I20250622 13:54:00.338249 5826 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 774f8894e66b49e49d745aa79721654b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 13:54:00.339486 5841 catalog_manager.cc:5582] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 reported cstate change: config changed from index -1 to 2, NON_VOTER ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } } }
I20250622 13:54:00.340041 6254 raft_consensus.cc:1062] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b: attempting to promote NON_VOTER ac521567b5ce4e179be2551ef1363f84 to VOTER
I20250622 13:54:00.340643 6254 consensus_queue.cc:237] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } }
I20250622 13:54:00.342613 6017 raft_consensus.cc:1273] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:54:00.343076 6223 raft_consensus.cc:1273] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 LEARNER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:54:00.343037 6280 consensus_queue.cc:1035] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:00.343588 6280 consensus_queue.cc:1035] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250622 13:54:00.344105 6057 consensus_peers.cc:487] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:00.346091 6263 raft_consensus.cc:2953] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:00.346665 6223 raft_consensus.cc:2953] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:00.349753 6017 raft_consensus.cc:2953] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:00.350423 5841 catalog_manager.cc:5582] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 reported cstate change: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:00.354561 6092 consensus_queue.cc:237] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } }
I20250622 13:54:00.356173 6017 raft_consensus.cc:1273] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250622 13:54:00.356173 6223 raft_consensus.cc:1273] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250622 13:54:00.356845 6254 consensus_queue.cc:1035] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:00.357206 6254 consensus_queue.cc:1035] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:00.358790 6280 raft_consensus.cc:2953] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:00.359335 6223 raft_consensus.cc:2953] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:00.359793 6017 raft_consensus.cc:2953] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:00.361771 5826 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 269be2c1778446f393fcca7c3314105e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:00.363386 5841 catalog_manager.cc:5582] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b reported cstate change: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250622 13:54:00.369035 5827 catalog_manager.cc:4671] TS 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 269be2c1778446f393fcca7c3314105e: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:54:00.514876 6291 ts_tablet_manager.cc:927] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: Initiating tablet copy from peer 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:54:00.515439 6291 tablet_copy_client.cc:323] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35437
I20250622 13:54:00.516026 6102 tablet_copy_service.cc:140] P 6946c521f740456dbe01946108d1986b: Received BeginTabletCopySession request for tablet f1a0591bf6b349dabdb48e2474adf87d from peer ac521567b5ce4e179be2551ef1363f84 ({username='slave'} at 127.0.0.1:52328)
I20250622 13:54:00.516185 6102 tablet_copy_service.cc:161] P 6946c521f740456dbe01946108d1986b: Beginning new tablet copy session on tablet f1a0591bf6b349dabdb48e2474adf87d from peer ac521567b5ce4e179be2551ef1363f84 at {username='slave'} at 127.0.0.1:52328: session id = ac521567b5ce4e179be2551ef1363f84-f1a0591bf6b349dabdb48e2474adf87d
I20250622 13:54:00.518177 6102 tablet_copy_source_session.cc:215] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:00.519212 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a0591bf6b349dabdb48e2474adf87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:00.523213 6291 tablet_copy_client.cc:806] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:00.523432 6291 tablet_copy_client.cc:670] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:00.524955 6291 tablet_copy_client.cc:538] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:00.527504 6291 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: Bootstrap starting.
I20250622 13:54:00.532001 6291 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:00.532375 6291 tablet_bootstrap.cc:492] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: Bootstrap complete.
I20250622 13:54:00.532600 6291 ts_tablet_manager.cc:1397] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250622 13:54:00.533216 6291 raft_consensus.cc:357] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:54:00.533424 6291 raft_consensus.cc:738] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ac521567b5ce4e179be2551ef1363f84, State: Initialized, Role: LEARNER
I20250622 13:54:00.533617 6291 consensus_queue.cc:260] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:54:00.534034 6291 ts_tablet_manager.cc:1428] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20250622 13:54:00.534832 6102 tablet_copy_service.cc:342] P 6946c521f740456dbe01946108d1986b: Request end of tablet copy session ac521567b5ce4e179be2551ef1363f84-f1a0591bf6b349dabdb48e2474adf87d received from {username='slave'} at 127.0.0.1:52328
I20250622 13:54:00.534971 6102 tablet_copy_service.cc:434] P 6946c521f740456dbe01946108d1986b: ending tablet copy session ac521567b5ce4e179be2551ef1363f84-f1a0591bf6b349dabdb48e2474adf87d on tablet f1a0591bf6b349dabdb48e2474adf87d with peer ac521567b5ce4e179be2551ef1363f84
I20250622 13:54:00.558877 6291 ts_tablet_manager.cc:927] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: Initiating tablet copy from peer 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:54:00.559414 6291 tablet_copy_client.cc:323] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35437
I20250622 13:54:00.560050 6102 tablet_copy_service.cc:140] P 6946c521f740456dbe01946108d1986b: Received BeginTabletCopySession request for tablet c01ef28c08a24ed48fa6cbb8d7af60bd from peer ac521567b5ce4e179be2551ef1363f84 ({username='slave'} at 127.0.0.1:52328)
I20250622 13:54:00.560185 6102 tablet_copy_service.cc:161] P 6946c521f740456dbe01946108d1986b: Beginning new tablet copy session on tablet c01ef28c08a24ed48fa6cbb8d7af60bd from peer ac521567b5ce4e179be2551ef1363f84 at {username='slave'} at 127.0.0.1:52328: session id = ac521567b5ce4e179be2551ef1363f84-c01ef28c08a24ed48fa6cbb8d7af60bd
I20250622 13:54:00.562433 6102 tablet_copy_source_session.cc:215] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:00.563434 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01ef28c08a24ed48fa6cbb8d7af60bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:00.567577 6291 tablet_copy_client.cc:806] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:00.567811 6291 tablet_copy_client.cc:670] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:00.569394 6291 tablet_copy_client.cc:538] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:00.571871 6291 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: Bootstrap starting.
I20250622 13:54:00.576265 6291 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:00.576622 6291 tablet_bootstrap.cc:492] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: Bootstrap complete.
I20250622 13:54:00.576854 6291 ts_tablet_manager.cc:1397] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250622 13:54:00.577484 6291 raft_consensus.cc:357] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:54:00.577661 6291 raft_consensus.cc:738] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ac521567b5ce4e179be2551ef1363f84, State: Initialized, Role: LEARNER
I20250622 13:54:00.577852 6291 consensus_queue.cc:260] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:54:00.578282 6291 ts_tablet_manager.cc:1428] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250622 13:54:00.579135 6102 tablet_copy_service.cc:342] P 6946c521f740456dbe01946108d1986b: Request end of tablet copy session ac521567b5ce4e179be2551ef1363f84-c01ef28c08a24ed48fa6cbb8d7af60bd received from {username='slave'} at 127.0.0.1:52328
I20250622 13:54:00.579274 6102 tablet_copy_service.cc:434] P 6946c521f740456dbe01946108d1986b: ending tablet copy session ac521567b5ce4e179be2551ef1363f84-c01ef28c08a24ed48fa6cbb8d7af60bd on tablet c01ef28c08a24ed48fa6cbb8d7af60bd with peer ac521567b5ce4e179be2551ef1363f84
W20250622 13:54:00.685686 5889 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1588f0f54ac744a489c91d50b6f692fd not available for placement
I20250622 13:54:00.786633 6291 ts_tablet_manager.cc:927] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: Initiating tablet copy from peer 6946c521f740456dbe01946108d1986b (127.0.103.67:35437)
I20250622 13:54:00.787156 6291 tablet_copy_client.cc:323] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35437
I20250622 13:54:00.787742 6102 tablet_copy_service.cc:140] P 6946c521f740456dbe01946108d1986b: Received BeginTabletCopySession request for tablet 774f8894e66b49e49d745aa79721654b from peer ac521567b5ce4e179be2551ef1363f84 ({username='slave'} at 127.0.0.1:52328)
I20250622 13:54:00.787882 6102 tablet_copy_service.cc:161] P 6946c521f740456dbe01946108d1986b: Beginning new tablet copy session on tablet 774f8894e66b49e49d745aa79721654b from peer ac521567b5ce4e179be2551ef1363f84 at {username='slave'} at 127.0.0.1:52328: session id = ac521567b5ce4e179be2551ef1363f84-774f8894e66b49e49d745aa79721654b
I20250622 13:54:00.789773 6102 tablet_copy_source_session.cc:215] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:00.790939 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 774f8894e66b49e49d745aa79721654b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:00.795233 6291 tablet_copy_client.cc:806] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:00.795485 6291 tablet_copy_client.cc:670] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:00.796727 6291 tablet_copy_client.cc:538] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:00.799014 6291 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: Bootstrap starting.
I20250622 13:54:00.803395 6291 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:00.803741 6291 tablet_bootstrap.cc:492] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: Bootstrap complete.
I20250622 13:54:00.803972 6291 ts_tablet_manager.cc:1397] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250622 13:54:00.804540 6291 raft_consensus.cc:357] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:54:00.804699 6291 raft_consensus.cc:738] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ac521567b5ce4e179be2551ef1363f84, State: Initialized, Role: LEARNER
I20250622 13:54:00.804879 6291 consensus_queue.cc:260] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: true } }
I20250622 13:54:00.805255 6291 ts_tablet_manager.cc:1428] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:54:00.806025 6102 tablet_copy_service.cc:342] P 6946c521f740456dbe01946108d1986b: Request end of tablet copy session ac521567b5ce4e179be2551ef1363f84-774f8894e66b49e49d745aa79721654b received from {username='slave'} at 127.0.0.1:52328
I20250622 13:54:00.806156 6102 tablet_copy_service.cc:434] P 6946c521f740456dbe01946108d1986b: ending tablet copy session ac521567b5ce4e179be2551ef1363f84-774f8894e66b49e49d745aa79721654b on tablet 774f8894e66b49e49d745aa79721654b with peer ac521567b5ce4e179be2551ef1363f84
I20250622 13:54:00.938706 6174 consensus_queue.cc:579] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Leader has been unable to successfully communicate with peer 1588f0f54ac744a489c91d50b6f692fd for more than 1 seconds (1.005s)
I20250622 13:54:01.105139 6223 raft_consensus.cc:1215] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250622 13:54:01.161374 6223 raft_consensus.cc:1215] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250622 13:54:01.306881 6298 consensus_queue.cc:579] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Leader has been unable to successfully communicate with peer 1588f0f54ac744a489c91d50b6f692fd for more than 1 seconds (1.370s)
I20250622 13:54:01.330840 6223 raft_consensus.cc:1215] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250622 13:54:01.549296 6298 raft_consensus.cc:1062] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b: attempting to promote NON_VOTER ac521567b5ce4e179be2551ef1363f84 to VOTER
I20250622 13:54:01.550040 6298 consensus_queue.cc:237] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } }
I20250622 13:54:01.552220 6223 raft_consensus.cc:1273] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:01.552704 6017 raft_consensus.cc:1273] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:01.552812 6174 consensus_queue.cc:1035] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250622 13:54:01.553148 5827 catalog_manager.cc:4671] TS 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 269be2c1778446f393fcca7c3314105e: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:54:01.553256 6298 consensus_queue.cc:1035] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250622 13:54:01.553694 6057 consensus_peers.cc:487] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:01.555542 6174 raft_consensus.cc:2953] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.556098 6017 raft_consensus.cc:2953] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.558966 6223 raft_consensus.cc:2953] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.559914 5839 catalog_manager.cc:5582] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b reported cstate change: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:54:01.564893 6092 consensus_queue.cc:237] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } }
I20250622 13:54:01.566444 6223 raft_consensus.cc:1273] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:01.566895 6304 consensus_queue.cc:1035] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:01.566949 6017 raft_consensus.cc:1273] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:01.567432 6304 consensus_queue.cc:1035] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:01.568787 6174 raft_consensus.cc:2953] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.569463 6223 raft_consensus.cc:2953] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.569962 6017 raft_consensus.cc:2953] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.572818 5826 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c01ef28c08a24ed48fa6cbb8d7af60bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:01.573058 5840 catalog_manager.cc:5582] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 reported cstate change: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
W20250622 13:54:01.686550 5889 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1588f0f54ac744a489c91d50b6f692fd not available for placement
I20250622 13:54:01.727193 6298 raft_consensus.cc:1062] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b: attempting to promote NON_VOTER ac521567b5ce4e179be2551ef1363f84 to VOTER
I20250622 13:54:01.727835 6298 consensus_queue.cc:237] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } }
I20250622 13:54:01.729677 6223 raft_consensus.cc:1273] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:01.729679 6017 raft_consensus.cc:1273] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:01.730244 6304 consensus_queue.cc:1035] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250622 13:54:01.730463 6057 consensus_peers.cc:487] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:01.730479 6298 consensus_queue.cc:1035] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:01.732800 6304 raft_consensus.cc:2953] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.733464 6223 raft_consensus.cc:2953] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.736279 6017 raft_consensus.cc:2953] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.736734 5839 catalog_manager.cc:5582] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b reported cstate change: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:54:01.740675 6092 consensus_queue.cc:237] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } }
I20250622 13:54:01.742369 6223 raft_consensus.cc:1273] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:01.742441 6017 raft_consensus.cc:1273] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:01.742839 6304 consensus_queue.cc:1035] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:01.743044 6174 consensus_queue.cc:1035] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:01.744737 6174 raft_consensus.cc:2953] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.745321 6017 raft_consensus.cc:2953] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.746294 6223 raft_consensus.cc:2953] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.748687 5826 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f1a0591bf6b349dabdb48e2474adf87d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:01.749032 5841 catalog_manager.cc:5582] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 reported cstate change: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.792575 6174 raft_consensus.cc:1062] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b: attempting to promote NON_VOTER ac521567b5ce4e179be2551ef1363f84 to VOTER
I20250622 13:54:01.793490 6174 consensus_queue.cc:237] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } }
I20250622 13:54:01.795699 6223 raft_consensus.cc:1273] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 LEARNER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:01.795707 6017 raft_consensus.cc:1273] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:01.796226 6298 consensus_queue.cc:1035] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250622 13:54:01.796730 6057 consensus_peers.cc:487] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b -> Peer 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): Couldn't send request to peer 1588f0f54ac744a489c91d50b6f692fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:01.796729 6304 consensus_queue.cc:1035] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:01.799853 6304 raft_consensus.cc:2953] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.800556 6223 raft_consensus.cc:2953] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.803246 6017 raft_consensus.cc:2953] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.803866 5840 catalog_manager.cc:5582] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b reported cstate change: config changed from index 2 to 3, ac521567b5ce4e179be2551ef1363f84 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1588f0f54ac744a489c91d50b6f692fd" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:54:01.808256 6092 consensus_queue.cc:237] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } }
I20250622 13:54:01.809885 6223 raft_consensus.cc:1273] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:01.810348 6174 consensus_queue.cc:1035] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:01.811097 6017 raft_consensus.cc:1273] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Refusing update from remote peer 6946c521f740456dbe01946108d1986b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:01.811519 6174 consensus_queue.cc:1035] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:01.812251 6263 raft_consensus.cc:2953] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.812847 6223 raft_consensus.cc:2953] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.814045 6017 raft_consensus.cc:2953] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } } }
I20250622 13:54:01.815845 5826 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 774f8894e66b49e49d745aa79721654b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:01.817262 5841 catalog_manager.cc:5582] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b reported cstate change: config changed from index 3 to 4, VOTER 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "6946c521f740456dbe01946108d1986b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6946c521f740456dbe01946108d1986b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5db8c0f5bd584a39a33146871659b264" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac521567b5ce4e179be2551ef1363f84" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 38175 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250622 13:54:02.584945 5827 catalog_manager.cc:4671] TS 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 269be2c1778446f393fcca7c3314105e: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
W20250622 13:54:03.789904 5827 catalog_manager.cc:4671] TS 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c01ef28c08a24ed48fa6cbb8d7af60bd: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
W20250622 13:54:04.015547 5827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20250622 13:54:04.695168 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:54:04.707407 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:04.707680 413 tablet_replica.cc:331] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264: stopping tablet replica
I20250622 13:54:04.707844 413 raft_consensus.cc:2241] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:04.708127 413 raft_consensus.cc:2270] T 774f8894e66b49e49d745aa79721654b P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.708693 413 tablet_replica.cc:331] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264: stopping tablet replica
I20250622 13:54:04.708846 413 raft_consensus.cc:2241] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:04.709059 413 raft_consensus.cc:2270] T f1a0591bf6b349dabdb48e2474adf87d P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.709539 413 tablet_replica.cc:331] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264: stopping tablet replica
I20250622 13:54:04.709682 413 raft_consensus.cc:2241] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:04.709896 413 raft_consensus.cc:2270] T c01ef28c08a24ed48fa6cbb8d7af60bd P 5db8c0f5bd584a39a33146871659b264 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.710321 413 tablet_replica.cc:331] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264: stopping tablet replica
I20250622 13:54:04.710460 413 raft_consensus.cc:2241] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:04.710666 413 raft_consensus.cc:2270] T 269be2c1778446f393fcca7c3314105e P 5db8c0f5bd584a39a33146871659b264 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250622 13:54:04.714497 6058 consensus_peers.cc:487] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b -> Peer 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955): Couldn't send request to peer 5db8c0f5bd584a39a33146871659b264. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:37955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:04.715137 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:54:04.718423 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
W20250622 13:54:04.723814 6058 consensus_peers.cc:487] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b -> Peer 5db8c0f5bd584a39a33146871659b264 (127.0.103.66:37955): Couldn't send request to peer 5db8c0f5bd584a39a33146871659b264. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:37955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:04.740459 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:04.740708 413 tablet_replica.cc:331] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b: stopping tablet replica
I20250622 13:54:04.740888 413 raft_consensus.cc:2241] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:54:04.741212 413 raft_consensus.cc:2270] T f1a0591bf6b349dabdb48e2474adf87d P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.741813 413 tablet_replica.cc:331] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b: stopping tablet replica
I20250622 13:54:04.741952 413 raft_consensus.cc:2241] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:54:04.742187 413 raft_consensus.cc:2270] T 774f8894e66b49e49d745aa79721654b P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.742604 413 tablet_replica.cc:331] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b: stopping tablet replica
I20250622 13:54:04.742728 413 raft_consensus.cc:2241] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:54:04.742961 413 raft_consensus.cc:2270] T c01ef28c08a24ed48fa6cbb8d7af60bd P 6946c521f740456dbe01946108d1986b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.743332 413 tablet_replica.cc:331] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b: stopping tablet replica
I20250622 13:54:04.743444 413 raft_consensus.cc:2241] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:04.743645 413 raft_consensus.cc:2270] T 269be2c1778446f393fcca7c3314105e P 6946c521f740456dbe01946108d1986b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.758201 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:54:04.761758 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250622 13:54:04.771169 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:04.771430 413 tablet_replica.cc:331] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84: stopping tablet replica
I20250622 13:54:04.771603 413 raft_consensus.cc:2241] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:04.771821 413 raft_consensus.cc:2270] T c01ef28c08a24ed48fa6cbb8d7af60bd P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.772399 413 tablet_replica.cc:331] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84: stopping tablet replica
I20250622 13:54:04.772524 413 raft_consensus.cc:2241] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:04.772706 413 raft_consensus.cc:2270] T 269be2c1778446f393fcca7c3314105e P ac521567b5ce4e179be2551ef1363f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.773119 413 tablet_replica.cc:331] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84: stopping tablet replica
I20250622 13:54:04.773267 413 raft_consensus.cc:2241] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:04.773470 413 raft_consensus.cc:2270] T 774f8894e66b49e49d745aa79721654b P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.773873 413 tablet_replica.cc:331] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84: stopping tablet replica
I20250622 13:54:04.773975 413 raft_consensus.cc:2241] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:04.774147 413 raft_consensus.cc:2270] T f1a0591bf6b349dabdb48e2474adf87d P ac521567b5ce4e179be2551ef1363f84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:04.787917 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250622 13:54:04.791384 413 master.cc:561] Master@127.0.103.126:39429 shutting down...
W20250622 13:54:05.875672 5827 catalog_manager.cc:4671] TS 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c01ef28c08a24ed48fa6cbb8d7af60bd: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
W20250622 13:54:08.793543 5827 catalog_manager.cc:4671] TS 1588f0f54ac744a489c91d50b6f692fd (127.0.103.65:38627): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 269be2c1778446f393fcca7c3314105e: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38627: connect: Connection refused (error 111)
I20250622 13:54:09.282428 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:09.282753 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:09.282888 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 99d072fed1d94108a1d8c3219baf9ab9: stopping tablet replica
I20250622 13:54:09.328994 413 master.cc:583] Master@127.0.103.126:39429 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13715 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20250622 13:54:09.343550 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:34635
I20250622 13:54:09.344103 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:09.346576 6326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:09.346706 6327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:09.347117 6329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:09.347285 413 server_base.cc:1048] running on GCE node
I20250622 13:54:09.347966 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:09.348050 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:09.348124 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600449348106 us; error 0 us; skew 500 ppm
I20250622 13:54:09.348297 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:09.354404 413 webserver.cc:469] Webserver started at http://127.0.103.126:36465/ using document root <none> and password file <none>
I20250622 13:54:09.354664 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:09.354744 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:09.354907 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:09.355576 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "71a871796b614b7e8b4ad5231cdb9747"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:09.358150 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:54:09.359793 6334 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.360196 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:54:09.360291 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "71a871796b614b7e8b4ad5231cdb9747"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:09.360399 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:09.377959 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:09.378453 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:09.390210 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:34635
I20250622 13:54:09.390214 6385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:34635 every 8 connection(s)
I20250622 13:54:09.391587 6386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:09.395742 6386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747: Bootstrap starting.
I20250622 13:54:09.397048 6386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:09.398646 6386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747: No bootstrap required, opened a new log
I20250622 13:54:09.399291 6386 raft_consensus.cc:357] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a871796b614b7e8b4ad5231cdb9747" member_type: VOTER }
I20250622 13:54:09.399425 6386 raft_consensus.cc:383] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:09.399499 6386 raft_consensus.cc:738] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a871796b614b7e8b4ad5231cdb9747, State: Initialized, Role: FOLLOWER
I20250622 13:54:09.399724 6386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a871796b614b7e8b4ad5231cdb9747" member_type: VOTER }
I20250622 13:54:09.399855 6386 raft_consensus.cc:397] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:54:09.399930 6386 raft_consensus.cc:491] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:54:09.400027 6386 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:09.401898 6386 raft_consensus.cc:513] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a871796b614b7e8b4ad5231cdb9747" member_type: VOTER }
I20250622 13:54:09.402086 6386 leader_election.cc:304] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71a871796b614b7e8b4ad5231cdb9747; no voters:
I20250622 13:54:09.402518 6386 leader_election.cc:290] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:54:09.402655 6389 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:09.403121 6389 raft_consensus.cc:695] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 1 LEADER]: Becoming Leader. State: Replica: 71a871796b614b7e8b4ad5231cdb9747, State: Running, Role: LEADER
I20250622 13:54:09.403365 6389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a871796b614b7e8b4ad5231cdb9747" member_type: VOTER }
I20250622 13:54:09.403532 6386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:54:09.404402 6390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71a871796b614b7e8b4ad5231cdb9747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a871796b614b7e8b4ad5231cdb9747" member_type: VOTER } }
I20250622 13:54:09.404457 6391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a871796b614b7e8b4ad5231cdb9747. Latest consensus state: current_term: 1 leader_uuid: "71a871796b614b7e8b4ad5231cdb9747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a871796b614b7e8b4ad5231cdb9747" member_type: VOTER } }
I20250622 13:54:09.404614 6390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [sys.catalog]: This master's current role is: LEADER
I20250622 13:54:09.404978 6391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [sys.catalog]: This master's current role is: LEADER
I20250622 13:54:09.408402 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20250622 13:54:09.409149 6405 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250622 13:54:09.409238 6405 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250622 13:54:09.409337 6396 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:54:09.410458 6396 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:54:09.413218 6396 catalog_manager.cc:1349] Generated new cluster ID: 245e6758e8944d41bb8e59b4b94bf65a
I20250622 13:54:09.413307 6396 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:54:09.421732 6396 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:54:09.422554 6396 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:54:09.429466 6396 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747: Generated new TSK 0
I20250622 13:54:09.429725 6396 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:54:09.441257 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:09.444011 6408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:09.444025 6407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:09.445287 6410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:09.445387 413 server_base.cc:1048] running on GCE node
I20250622 13:54:09.445700 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:09.445777 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:09.445843 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600449445821 us; error 0 us; skew 500 ppm
I20250622 13:54:09.445996 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:09.447292 413 webserver.cc:469] Webserver started at http://127.0.103.65:35623/ using document root <none> and password file <none>
I20250622 13:54:09.447511 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:09.447606 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:09.447743 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:09.448364 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "b6a4d13e04284fd883752fd0449a6f28"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:09.460757 413 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.009s sys 0.004s
I20250622 13:54:09.462539 6415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.462949 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250622 13:54:09.463052 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "b6a4d13e04284fd883752fd0449a6f28"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:09.463188 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:09.471081 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:09.471535 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:09.472052 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:09.472848 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:54:09.472903 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.472985 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:54:09.473022 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.484228 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38653
I20250622 13:54:09.484246 6477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38653 every 8 connection(s)
I20250622 13:54:09.485788 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:09.488663 6482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:09.488689 6483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:09.490352 413 server_base.cc:1048] running on GCE node
W20250622 13:54:09.490516 6485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:09.490861 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:09.490921 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:09.490983 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600449490978 us; error 0 us; skew 500 ppm
I20250622 13:54:09.491124 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:09.492575 413 webserver.cc:469] Webserver started at http://127.0.103.66:34665/ using document root <none> and password file <none>
I20250622 13:54:09.492798 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:09.492877 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:09.492998 6478 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34635
I20250622 13:54:09.493003 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:09.493131 6478 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:09.493538 6478 heartbeater.cc:507] Master 127.0.103.126:34635 requested a full tablet report, sending...
I20250622 13:54:09.493834 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "91525564458a4fa8acba6cbef9652697"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:09.494392 6351 ts_manager.cc:194] Registered new tserver with Master: b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653)
I20250622 13:54:09.495394 6351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56078
I20250622 13:54:09.496279 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:54:09.497778 6490 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.498104 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:09.498196 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "91525564458a4fa8acba6cbef9652697"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:09.498289 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:09.518442 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:09.518982 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:09.519574 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:09.520416 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:54:09.520473 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.520550 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:54:09.520587 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.531952 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:41775
I20250622 13:54:09.531973 6552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:41775 every 8 connection(s)
I20250622 13:54:09.533520 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:09.536005 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:09.536387 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:09.536375 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:09.537505 413 server_base.cc:1048] running on GCE node
I20250622 13:54:09.540588 6553 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34635
I20250622 13:54:09.540746 6553 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:09.540864 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:09.540940 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:09.540987 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600449540982 us; error 0 us; skew 500 ppm
I20250622 13:54:09.541106 6553 heartbeater.cc:507] Master 127.0.103.126:34635 requested a full tablet report, sending...
I20250622 13:54:09.541124 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:09.542016 6351 ts_manager.cc:194] Registered new tserver with Master: 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
I20250622 13:54:09.542466 413 webserver.cc:469] Webserver started at http://127.0.103.67:37245/ using document root <none> and password file <none>
I20250622 13:54:09.542693 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:09.542764 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:09.542877 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:09.543035 6351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56080
I20250622 13:54:09.543589 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "9f774500db5f4d6f912457b674f9fcb7"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:09.545882 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250622 13:54:09.547341 6564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.547652 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250622 13:54:09.547731 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "9f774500db5f4d6f912457b674f9fcb7"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:09.547827 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:09.579165 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:09.579643 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:09.580159 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:09.580971 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:54:09.581023 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.581076 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:54:09.581120 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:09.593915 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:37405
I20250622 13:54:09.593950 6626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:37405 every 8 connection(s)
I20250622 13:54:09.600123 6627 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34635
I20250622 13:54:09.600250 6627 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:09.600538 6627 heartbeater.cc:507] Master 127.0.103.126:34635 requested a full tablet report, sending...
I20250622 13:54:09.601380 6351 ts_manager.cc:194] Registered new tserver with Master: 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:09.601576 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006794684s
I20250622 13:54:09.602121 6351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56092
I20250622 13:54:10.496614 6478 heartbeater.cc:499] Master 127.0.103.126:34635 was elected leader, sending a full tablet report...
I20250622 13:54:10.544168 6553 heartbeater.cc:499] Master 127.0.103.126:34635 was elected leader, sending a full tablet report...
I20250622 13:54:10.603367 6627 heartbeater.cc:499] Master 127.0.103.126:34635 was elected leader, sending a full tablet report...
I20250622 13:54:10.629513 413 test_util.cc:276] Using random seed: 1845550980
I20250622 13:54:10.641620 6351 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56096:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:54:10.642319 6351 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:54:10.667722 6518 tablet_service.cc:1468] Processing CreateTablet for tablet c9994feeb6d842f694ef146799ba1e38 (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:10.668304 6518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9994feeb6d842f694ef146799ba1e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.668258 6516 tablet_service.cc:1468] Processing CreateTablet for tablet acec09482db244de9e566ebc4e0ef93c (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:10.668668 6516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acec09482db244de9e566ebc4e0ef93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.667819 6517 tablet_service.cc:1468] Processing CreateTablet for tablet 1414acbf4f0446ab8a76fac5307f41fd (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:10.669145 6517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1414acbf4f0446ab8a76fac5307f41fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.669368 6592 tablet_service.cc:1468] Processing CreateTablet for tablet c9994feeb6d842f694ef146799ba1e38 (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:10.669731 6515 tablet_service.cc:1468] Processing CreateTablet for tablet 9a99088323b24b71ac515b13676963a8 (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:10.669880 6592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9994feeb6d842f694ef146799ba1e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.670087 6515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a99088323b24b71ac515b13676963a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.673758 6590 tablet_service.cc:1468] Processing CreateTablet for tablet acec09482db244de9e566ebc4e0ef93c (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:10.674258 6590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acec09482db244de9e566ebc4e0ef93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.674213 6589 tablet_service.cc:1468] Processing CreateTablet for tablet 9a99088323b24b71ac515b13676963a8 (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:10.674885 6589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a99088323b24b71ac515b13676963a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.676996 6591 tablet_service.cc:1468] Processing CreateTablet for tablet 1414acbf4f0446ab8a76fac5307f41fd (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:10.677461 6591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1414acbf4f0446ab8a76fac5307f41fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.683336 6443 tablet_service.cc:1468] Processing CreateTablet for tablet c9994feeb6d842f694ef146799ba1e38 (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:10.683789 6443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9994feeb6d842f694ef146799ba1e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.684384 6442 tablet_service.cc:1468] Processing CreateTablet for tablet 1414acbf4f0446ab8a76fac5307f41fd (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:10.684968 6442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1414acbf4f0446ab8a76fac5307f41fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.693063 6442 tablet_service.cc:1468] Processing CreateTablet for tablet acec09482db244de9e566ebc4e0ef93c (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:10.693274 6647 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Bootstrap starting.
I20250622 13:54:10.693708 6442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acec09482db244de9e566ebc4e0ef93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.694698 6441 tablet_service.cc:1468] Processing CreateTablet for tablet 9a99088323b24b71ac515b13676963a8 (DEFAULT_TABLE table=test-workload [id=2d7ecd54faad4dfdac3025b990a83895]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:10.695195 6441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a99088323b24b71ac515b13676963a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:10.695521 6647 tablet_bootstrap.cc:654] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.698770 6649 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Bootstrap starting.
I20250622 13:54:10.701318 6649 tablet_bootstrap.cc:654] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.703490 6647 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: No bootstrap required, opened a new log
I20250622 13:54:10.703701 6647 ts_tablet_manager.cc:1397] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250622 13:54:10.704340 6649 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: No bootstrap required, opened a new log
I20250622 13:54:10.704540 6647 raft_consensus.cc:357] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.704804 6647 raft_consensus.cc:383] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.704658 6649 ts_tablet_manager.cc:1397] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250622 13:54:10.704945 6647 raft_consensus.cc:738] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.705320 6647 consensus_queue.cc:260] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.705937 6649 raft_consensus.cc:357] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.706219 6649 raft_consensus.cc:383] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.706368 6649 raft_consensus.cc:738] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.706676 6649 consensus_queue.cc:260] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.711582 6648 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap starting.
I20250622 13:54:10.712217 6649 ts_tablet_manager.cc:1428] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250622 13:54:10.713454 6647 ts_tablet_manager.cc:1428] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250622 13:54:10.713495 6649 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Bootstrap starting.
I20250622 13:54:10.715008 6649 tablet_bootstrap.cc:654] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.715178 6648 tablet_bootstrap.cc:654] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.716418 6647 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Bootstrap starting.
I20250622 13:54:10.717861 6648 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: No bootstrap required, opened a new log
I20250622 13:54:10.717916 6649 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: No bootstrap required, opened a new log
I20250622 13:54:10.718084 6647 tablet_bootstrap.cc:654] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.718184 6649 ts_tablet_manager.cc:1397] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250622 13:54:10.718442 6648 ts_tablet_manager.cc:1397] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20250622 13:54:10.719095 6649 raft_consensus.cc:357] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.719218 6648 raft_consensus.cc:357] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.719317 6649 raft_consensus.cc:383] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.719429 6648 raft_consensus.cc:383] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.719496 6649 raft_consensus.cc:738] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.719524 6648 raft_consensus.cc:738] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.719744 6649 consensus_queue.cc:260] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.719807 6647 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: No bootstrap required, opened a new log
I20250622 13:54:10.719802 6648 consensus_queue.cc:260] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.720007 6647 ts_tablet_manager.cc:1397] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:54:10.720290 6649 ts_tablet_manager.cc:1428] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:10.720705 6649 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Bootstrap starting.
I20250622 13:54:10.721131 6647 raft_consensus.cc:357] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.721340 6647 raft_consensus.cc:383] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.721500 6647 raft_consensus.cc:738] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.721791 6647 consensus_queue.cc:260] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.722075 6649 tablet_bootstrap.cc:654] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.722385 6647 ts_tablet_manager.cc:1428] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:54:10.722700 6647 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Bootstrap starting.
I20250622 13:54:10.724733 6647 tablet_bootstrap.cc:654] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.730100 6648 ts_tablet_manager.cc:1428] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Time spent starting tablet: real 0.011s user 0.000s sys 0.008s
I20250622 13:54:10.730413 6648 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap starting.
I20250622 13:54:10.732267 6648 tablet_bootstrap.cc:654] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.732580 6647 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: No bootstrap required, opened a new log
I20250622 13:54:10.732780 6647 ts_tablet_manager.cc:1397] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20250622 13:54:10.734025 6649 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: No bootstrap required, opened a new log
I20250622 13:54:10.734182 6649 ts_tablet_manager.cc:1397] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250622 13:54:10.734866 6649 raft_consensus.cc:357] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.735059 6649 raft_consensus.cc:383] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.735229 6649 raft_consensus.cc:738] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.735028 6647 raft_consensus.cc:357] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.735347 6647 raft_consensus.cc:383] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.735458 6647 raft_consensus.cc:738] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.735514 6649 consensus_queue.cc:260] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.735734 6647 consensus_queue.cc:260] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.736002 6649 ts_tablet_manager.cc:1428] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250622 13:54:10.736354 6479 tablet.cc:2378] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:54:10.736351 6649 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Bootstrap starting.
I20250622 13:54:10.736230 6647 ts_tablet_manager.cc:1428] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250622 13:54:10.737012 6647 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Bootstrap starting.
I20250622 13:54:10.737918 6649 tablet_bootstrap.cc:654] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.738596 6647 tablet_bootstrap.cc:654] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.743526 6647 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: No bootstrap required, opened a new log
I20250622 13:54:10.743619 6649 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: No bootstrap required, opened a new log
I20250622 13:54:10.743695 6647 ts_tablet_manager.cc:1397] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20250622 13:54:10.743781 6649 ts_tablet_manager.cc:1397] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250622 13:54:10.744524 6649 raft_consensus.cc:357] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.744524 6647 raft_consensus.cc:357] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.744750 6649 raft_consensus.cc:383] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.744760 6647 raft_consensus.cc:383] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.744861 6649 raft_consensus.cc:738] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.744861 6647 raft_consensus.cc:738] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.745136 6649 consensus_queue.cc:260] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.745139 6647 consensus_queue.cc:260] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.745733 6649 ts_tablet_manager.cc:1428] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250622 13:54:10.746361 6647 ts_tablet_manager.cc:1428] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250622 13:54:10.746860 6648 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: No bootstrap required, opened a new log
I20250622 13:54:10.747108 6648 ts_tablet_manager.cc:1397] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.009s
I20250622 13:54:10.747942 6648 raft_consensus.cc:357] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.748276 6648 raft_consensus.cc:383] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.748378 6648 raft_consensus.cc:738] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.748657 6648 consensus_queue.cc:260] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.749152 6648 ts_tablet_manager.cc:1428] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250622 13:54:10.749509 6648 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap starting.
I20250622 13:54:10.751017 6648 tablet_bootstrap.cc:654] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.752480 6648 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: No bootstrap required, opened a new log
I20250622 13:54:10.752593 6648 ts_tablet_manager.cc:1397] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:54:10.753108 6648 raft_consensus.cc:357] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.753249 6648 raft_consensus.cc:383] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.753302 6648 raft_consensus.cc:738] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.753521 6648 consensus_queue.cc:260] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.753882 6648 ts_tablet_manager.cc:1428] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250622 13:54:10.754179 6648 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap starting.
I20250622 13:54:10.755239 6648 tablet_bootstrap.cc:654] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:10.756641 6648 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: No bootstrap required, opened a new log
I20250622 13:54:10.756753 6648 ts_tablet_manager.cc:1397] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250622 13:54:10.757273 6648 raft_consensus.cc:357] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.757455 6648 raft_consensus.cc:383] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:10.757515 6648 raft_consensus.cc:738] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Initialized, Role: FOLLOWER
I20250622 13:54:10.757683 6648 consensus_queue.cc:260] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.757997 6648 ts_tablet_manager.cc:1428] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:10.774487 6653 raft_consensus.cc:491] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:10.774684 6653 raft_consensus.cc:513] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.775497 6653 leader_election.cc:290] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:10.783063 6528 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1414acbf4f0446ab8a76fac5307f41fd" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697" is_pre_election: true
I20250622 13:54:10.783064 6602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1414acbf4f0446ab8a76fac5307f41fd" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:10.783350 6528 raft_consensus.cc:2466] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 0.
I20250622 13:54:10.783371 6602 raft_consensus.cc:2466] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 0.
I20250622 13:54:10.783809 6417 leader_election.cc:304] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:10.784116 6653 raft_consensus.cc:2802] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:10.784204 6653 raft_consensus.cc:491] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:10.784307 6653 raft_consensus.cc:3058] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.786257 6653 raft_consensus.cc:513] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.786789 6653 leader_election.cc:290] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 election: Requested vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:10.787220 6528 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1414acbf4f0446ab8a76fac5307f41fd" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697"
I20250622 13:54:10.787216 6602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1414acbf4f0446ab8a76fac5307f41fd" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:10.787410 6528 raft_consensus.cc:3058] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.787410 6602 raft_consensus.cc:3058] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.789476 6528 raft_consensus.cc:2466] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:10.789515 6602 raft_consensus.cc:2466] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:10.789842 6417 leader_election.cc:304] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:10.790094 6653 raft_consensus.cc:2802] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:10.790477 6653 raft_consensus.cc:695] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: Becoming Leader. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Running, Role: LEADER
I20250622 13:54:10.790784 6653 consensus_queue.cc:237] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.793107 6350 catalog_manager.cc:5582] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:10.805640 6653 raft_consensus.cc:491] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:10.805852 6653 raft_consensus.cc:513] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.806627 6653 leader_election.cc:290] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:10.806936 6528 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697" is_pre_election: true
I20250622 13:54:10.807029 6602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:10.807132 6528 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 0.
I20250622 13:54:10.807229 6602 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 0.
I20250622 13:54:10.807546 6417 leader_election.cc:304] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:10.807829 6653 raft_consensus.cc:2802] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:10.807906 6653 raft_consensus.cc:491] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:10.807974 6653 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.809898 6653 raft_consensus.cc:513] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.810438 6653 leader_election.cc:290] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 election: Requested vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:10.810757 6528 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697"
I20250622 13:54:10.810839 6602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:10.810933 6528 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.811014 6602 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.813032 6528 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:10.813117 6602 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:10.813432 6417 leader_election.cc:304] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:10.813692 6653 raft_consensus.cc:2802] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:10.813838 6653 raft_consensus.cc:695] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: Becoming Leader. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Running, Role: LEADER
I20250622 13:54:10.814136 6653 consensus_queue.cc:237] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.816520 6350 catalog_manager.cc:5582] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:10.893245 6657 raft_consensus.cc:491] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:10.893499 6657 raft_consensus.cc:513] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.894351 6657 leader_election.cc:290] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653), 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
I20250622 13:54:10.900878 6453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a99088323b24b71ac515b13676963a8" candidate_uuid: "9f774500db5f4d6f912457b674f9fcb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a4d13e04284fd883752fd0449a6f28" is_pre_election: true
I20250622 13:54:10.901021 6528 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a99088323b24b71ac515b13676963a8" candidate_uuid: "9f774500db5f4d6f912457b674f9fcb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697" is_pre_election: true
I20250622 13:54:10.901111 6453 raft_consensus.cc:2466] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f774500db5f4d6f912457b674f9fcb7 in term 0.
I20250622 13:54:10.901242 6528 raft_consensus.cc:2466] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f774500db5f4d6f912457b674f9fcb7 in term 0.
I20250622 13:54:10.901538 6566 leader_election.cc:304] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f774500db5f4d6f912457b674f9fcb7, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:10.901775 6657 raft_consensus.cc:2802] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:10.901862 6657 raft_consensus.cc:491] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:10.901950 6657 raft_consensus.cc:3058] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.904286 6657 raft_consensus.cc:513] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.904861 6657 leader_election.cc:290] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [CANDIDATE]: Term 1 election: Requested vote from peers b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653), 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
I20250622 13:54:10.905244 6453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a99088323b24b71ac515b13676963a8" candidate_uuid: "9f774500db5f4d6f912457b674f9fcb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a4d13e04284fd883752fd0449a6f28"
I20250622 13:54:10.905270 6528 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a99088323b24b71ac515b13676963a8" candidate_uuid: "9f774500db5f4d6f912457b674f9fcb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697"
I20250622 13:54:10.905421 6453 raft_consensus.cc:3058] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.905462 6528 raft_consensus.cc:3058] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.907553 6453 raft_consensus.cc:2466] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f774500db5f4d6f912457b674f9fcb7 in term 1.
I20250622 13:54:10.907549 6528 raft_consensus.cc:2466] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f774500db5f4d6f912457b674f9fcb7 in term 1.
I20250622 13:54:10.907949 6566 leader_election.cc:304] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f774500db5f4d6f912457b674f9fcb7, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:10.908211 6657 raft_consensus.cc:2802] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:10.908615 6657 raft_consensus.cc:695] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 LEADER]: Becoming Leader. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Running, Role: LEADER
I20250622 13:54:10.908857 6657 consensus_queue.cc:237] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:10.911185 6350 catalog_manager.cc:5582] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9f774500db5f4d6f912457b674f9fcb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: HEALTHY } } }
I20250622 13:54:10.973454 6652 raft_consensus.cc:491] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:10.973690 6652 raft_consensus.cc:513] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.974474 6652 leader_election.cc:290] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:10.977795 6653 raft_consensus.cc:491] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:10.977957 6653 raft_consensus.cc:513] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.978622 6653 leader_election.cc:290] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405), 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
I20250622 13:54:10.979118 6602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:10.979235 6528 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697" is_pre_election: true
I20250622 13:54:10.979357 6602 raft_consensus.cc:2466] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 0.
I20250622 13:54:10.979472 6528 raft_consensus.cc:2466] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 0.
I20250622 13:54:10.979843 6419 leader_election.cc:304] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f774500db5f4d6f912457b674f9fcb7, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:10.980150 6653 raft_consensus.cc:2802] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:10.980271 6653 raft_consensus.cc:491] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:10.980376 6653 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.982450 6602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:10.982664 6602 raft_consensus.cc:2466] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91525564458a4fa8acba6cbef9652697 in term 0.
I20250622 13:54:10.982733 6453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a4d13e04284fd883752fd0449a6f28" is_pre_election: true
I20250622 13:54:10.983008 6494 leader_election.cc:304] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, 9f774500db5f4d6f912457b674f9fcb7; no voters:
I20250622 13:54:10.983289 6652 raft_consensus.cc:2802] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:10.983402 6652 raft_consensus.cc:491] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:10.983507 6652 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.983572 6653 raft_consensus.cc:513] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.983927 6453 raft_consensus.cc:2391] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91525564458a4fa8acba6cbef9652697 in current term 1: Already voted for candidate b6a4d13e04284fd883752fd0449a6f28 in this term.
I20250622 13:54:10.984602 6602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:10.984616 6528 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697"
I20250622 13:54:10.984789 6602 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:10.984915 6653 leader_election.cc:290] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 election: Requested vote from peers 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405), 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
I20250622 13:54:10.986291 6652 raft_consensus.cc:513] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.986616 6528 raft_consensus.cc:2391] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b6a4d13e04284fd883752fd0449a6f28 in current term 1: Already voted for candidate 91525564458a4fa8acba6cbef9652697 in this term.
I20250622 13:54:10.987010 6602 raft_consensus.cc:2466] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:10.987365 6419 leader_election.cc:304] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f774500db5f4d6f912457b674f9fcb7, b6a4d13e04284fd883752fd0449a6f28; no voters: 91525564458a4fa8acba6cbef9652697
I20250622 13:54:10.987437 6601 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:10.987720 6601 raft_consensus.cc:2391] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 91525564458a4fa8acba6cbef9652697 in current term 1: Already voted for candidate b6a4d13e04284fd883752fd0449a6f28 in this term.
I20250622 13:54:10.987674 6453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a4d13e04284fd883752fd0449a6f28"
I20250622 13:54:10.987870 6453 raft_consensus.cc:2391] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 91525564458a4fa8acba6cbef9652697 in current term 1: Already voted for candidate b6a4d13e04284fd883752fd0449a6f28 in this term.
I20250622 13:54:10.987941 6652 leader_election.cc:290] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 1 election: Requested vote from peers b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:10.988099 6653 raft_consensus.cc:2802] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:10.988252 6653 raft_consensus.cc:695] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: Becoming Leader. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Running, Role: LEADER
I20250622 13:54:10.988219 6492 leader_election.cc:304] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91525564458a4fa8acba6cbef9652697; no voters: 9f774500db5f4d6f912457b674f9fcb7, b6a4d13e04284fd883752fd0449a6f28
I20250622 13:54:10.988507 6653 consensus_queue.cc:237] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:10.988677 6652 raft_consensus.cc:2747] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250622 13:54:10.990938 6350 catalog_manager.cc:5582] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:11.031972 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:54:11.039557 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:11.039952 413 tablet_replica.cc:331] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: stopping tablet replica
I20250622 13:54:11.040156 413 raft_consensus.cc:2241] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:11.040342 413 raft_consensus.cc:2270] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.041115 413 tablet_replica.cc:331] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: stopping tablet replica
I20250622 13:54:11.041265 413 raft_consensus.cc:2241] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:11.041504 413 pending_rounds.cc:70] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Trying to abort 1 pending ops.
I20250622 13:54:11.041571 413 pending_rounds.cc:77] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459446537019392 op_type: NO_OP noop_request { }
I20250622 13:54:11.041703 413 raft_consensus.cc:2887] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:54:11.041812 413 raft_consensus.cc:2270] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.042519 413 tablet_replica.cc:331] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: stopping tablet replica
I20250622 13:54:11.042667 413 raft_consensus.cc:2241] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:11.042860 413 pending_rounds.cc:70] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Trying to abort 1 pending ops.
I20250622 13:54:11.042922 413 pending_rounds.cc:77] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459447251091456 op_type: NO_OP noop_request { }
I20250622 13:54:11.043026 413 raft_consensus.cc:2887] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:54:11.043131 413 raft_consensus.cc:2270] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.043787 413 tablet_replica.cc:331] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: stopping tablet replica
I20250622 13:54:11.043936 413 raft_consensus.cc:2241] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:11.044116 413 pending_rounds.cc:70] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Trying to abort 1 pending ops.
I20250622 13:54:11.044176 413 pending_rounds.cc:77] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459446441525248 op_type: NO_OP noop_request { }
I20250622 13:54:11.044277 413 raft_consensus.cc:2887] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:54:11.044377 413 raft_consensus.cc:2270] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.049504 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250622 13:54:11.052953 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:54:11.061873 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:11.063131 413 tablet_replica.cc:331] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: stopping tablet replica
I20250622 13:54:11.063351 413 raft_consensus.cc:2241] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:11.063557 413 raft_consensus.cc:2270] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.064208 413 tablet_replica.cc:331] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: stopping tablet replica
I20250622 13:54:11.064365 413 raft_consensus.cc:2241] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:11.064512 413 raft_consensus.cc:2270] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.064994 413 tablet_replica.cc:331] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: stopping tablet replica
I20250622 13:54:11.065129 413 raft_consensus.cc:2241] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:11.065258 413 raft_consensus.cc:2270] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.065747 413 tablet_replica.cc:331] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: stopping tablet replica
I20250622 13:54:11.065872 413 raft_consensus.cc:2241] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:11.066004 413 raft_consensus.cc:2270] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.086428 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:54:11.089841 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:54:11.098516 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:11.098802 413 tablet_replica.cc:331] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: stopping tablet replica
I20250622 13:54:11.099036 413 raft_consensus.cc:2241] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:11.099190 413 raft_consensus.cc:2270] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.099750 413 tablet_replica.cc:331] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: stopping tablet replica
I20250622 13:54:11.099859 413 raft_consensus.cc:2241] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:11.099953 413 raft_consensus.cc:2270] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.100381 413 tablet_replica.cc:331] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: stopping tablet replica
I20250622 13:54:11.100474 413 raft_consensus.cc:2241] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:11.100562 413 raft_consensus.cc:2270] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.100986 413 tablet_replica.cc:331] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: stopping tablet replica
I20250622 13:54:11.101071 413 raft_consensus.cc:2241] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:11.101233 413 pending_rounds.cc:70] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Trying to abort 1 pending ops.
I20250622 13:54:11.101266 413 pending_rounds.cc:77] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459446924947456 op_type: NO_OP noop_request { }
I20250622 13:54:11.101378 413 raft_consensus.cc:2887] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:54:11.101452 413 raft_consensus.cc:2270] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:11.105862 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250622 13:54:11.109915 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:11.112695 6677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:11.114193 6678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:11.114385 6680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:11.114457 413 server_base.cc:1048] running on GCE node
I20250622 13:54:11.114789 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:11.114877 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:11.114948 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600451114944 us; error 0 us; skew 500 ppm
I20250622 13:54:11.115113 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:11.116448 413 webserver.cc:469] Webserver started at http://127.0.103.68:33831/ using document root <none> and password file <none>
I20250622 13:54:11.116695 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:11.116796 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:11.117043 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:11.117955 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-3-root/instance:
uuid: "5636f6d75d3546248fbe04dd72546026"
format_stamp: "Formatted at 2025-06-22 13:54:11 on dist-test-slave-tmv0"
I20250622 13:54:11.120999 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250622 13:54:11.122846 6685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:11.123313 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:11.123438 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-3-root
uuid: "5636f6d75d3546248fbe04dd72546026"
format_stamp: "Formatted at 2025-06-22 13:54:11 on dist-test-slave-tmv0"
I20250622 13:54:11.123548 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:11.146843 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:11.147424 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:11.147974 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:11.148918 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:54:11.148990 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:11.149091 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:54:11.149159 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:11.161747 6747 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:35959 every 8 connection(s)
I20250622 13:54:11.161742 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:35959
I20250622 13:54:11.168668 6748 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34635
I20250622 13:54:11.168833 6748 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:11.169243 6748 heartbeater.cc:507] Master 127.0.103.126:34635 requested a full tablet report, sending...
I20250622 13:54:11.170032 6350 ts_manager.cc:194] Registered new tserver with Master: 5636f6d75d3546248fbe04dd72546026 (127.0.103.68:35959)
I20250622 13:54:11.170765 6350 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56098
W20250622 13:54:11.410750 6399 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:11.411914 6338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250622 13:54:11.413060 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:11.413167 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:11.414657 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:11.414731 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:11.415957 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:11.416028 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
I20250622 13:54:12.172025 6748 heartbeater.cc:499] Master 127.0.103.126:34635 was elected leader, sending a full tablet report...
W20250622 13:54:12.420090 6399 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:12.425899 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:12.426003 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:12.427682 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:12.427764 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:12.429128 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:12.429200 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:13.433020 6399 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:13.441566 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:13.441685 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:13.443292 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:13.443375 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:13.444769 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:13.444837 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:14.448542 6399 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:14.454348 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:14.454440 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:14.455865 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:14.455937 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:14.457185 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:14.457244 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:15.460841 6399 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:15.466444 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:15.466534 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:15.467962 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:15.468027 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:37405: connect: Connection refused (error 111)
W20250622 13:54:15.469257 6399 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
W20250622 13:54:15.469316 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38653: connect: Connection refused (error 111)
I20250622 13:54:16.428800 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:16.431705 6767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:16.432066 6768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:16.432353 6770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:16.440750 413 server_base.cc:1048] running on GCE node
I20250622 13:54:16.441951 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:16.442018 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:16.442077 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600456442049 us; error 0 us; skew 500 ppm
I20250622 13:54:16.442245 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:16.449580 413 webserver.cc:469] Webserver started at http://127.0.103.65:35623/ using document root <none> and password file <none>
I20250622 13:54:16.449887 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:16.449977 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250622 13:54:16.452019 413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20250622 13:54:16.453760 6775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:16.454221 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:54:16.454339 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "b6a4d13e04284fd883752fd0449a6f28"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:16.454474 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:16.468536 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:16.469107 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:16.469655 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20250622 13:54:16.470206 6399 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver b6a4d13e04284fd883752fd0449a6f28 not available for placement
I20250622 13:54:16.471228 6782 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
I20250622 13:54:16.479075 413 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250622 13:54:16.479178 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20250622 13:54:16.479292 413 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20250622 13:54:16.481396 6782 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Bootstrap starting.
I20250622 13:54:16.485566 6782 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250622 13:54:16.486130 6782 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Bootstrap complete.
I20250622 13:54:16.486377 6782 ts_tablet_manager.cc:1397] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20250622 13:54:16.486886 413 ts_tablet_manager.cc:610] Registered 4 tablets
I20250622 13:54:16.486966 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.003s sys 0.004s
I20250622 13:54:16.487037 6782 raft_consensus.cc:357] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.487335 6782 raft_consensus.cc:738] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.487648 6782 consensus_queue.cc:260] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.488303 6782 ts_tablet_manager.cc:1428] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250622 13:54:16.488588 6782 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Bootstrap starting.
I20250622 13:54:16.492791 6782 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250622 13:54:16.493284 6782 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Bootstrap complete.
I20250622 13:54:16.493613 6782 ts_tablet_manager.cc:1397] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250622 13:54:16.494269 6782 raft_consensus.cc:357] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:16.494601 6782 raft_consensus.cc:738] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.494879 6782 consensus_queue.cc:260] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:16.495354 6782 ts_tablet_manager.cc:1428] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:16.495592 6782 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Bootstrap starting.
I20250622 13:54:16.499050 6782 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:16.499398 6782 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Bootstrap complete.
I20250622 13:54:16.499646 6782 ts_tablet_manager.cc:1397] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:54:16.500293 6782 raft_consensus.cc:357] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.500459 6782 raft_consensus.cc:738] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.500813 6782 consensus_queue.cc:260] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.501283 6782 ts_tablet_manager.cc:1428] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:54:16.501586 6782 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Bootstrap starting.
I20250622 13:54:16.505388 6782 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250622 13:54:16.505896 6782 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Bootstrap complete.
I20250622 13:54:16.506146 6782 ts_tablet_manager.cc:1397] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250622 13:54:16.506800 6782 raft_consensus.cc:357] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.507028 6782 raft_consensus.cc:738] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.507272 6782 consensus_queue.cc:260] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.507725 6782 ts_tablet_manager.cc:1428] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:54:16.511617 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38653
I20250622 13:54:16.511638 6843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38653 every 8 connection(s)
I20250622 13:54:16.513020 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:16.515827 6848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:16.518813 6849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:16.520081 6851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:16.520498 413 server_base.cc:1048] running on GCE node
I20250622 13:54:16.520831 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:16.520923 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:16.520990 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600456520963 us; error 0 us; skew 500 ppm
I20250622 13:54:16.521147 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:16.522774 413 webserver.cc:469] Webserver started at http://127.0.103.66:34665/ using document root <none> and password file <none>
I20250622 13:54:16.523092 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:16.523198 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250622 13:54:16.525601 6844 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34635
I20250622 13:54:16.525753 6844 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:16.525879 413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20250622 13:54:16.526152 6844 heartbeater.cc:507] Master 127.0.103.126:34635 requested a full tablet report, sending...
I20250622 13:54:16.527519 6350 ts_manager.cc:194] Re-registered known tserver with Master: b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653)
I20250622 13:54:16.527680 6856 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:16.528059 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250622 13:54:16.528187 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "91525564458a4fa8acba6cbef9652697"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:16.528309 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:16.529851 6350 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45676
I20250622 13:54:16.538475 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:16.539012 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:16.539527 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:16.548213 413 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250622 13:54:16.548306 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20250622 13:54:16.550307 6863 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Bootstrap starting.
I20250622 13:54:16.554044 6863 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:16.554486 6863 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Bootstrap complete.
I20250622 13:54:16.554714 6863 ts_tablet_manager.cc:1397] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250622 13:54:16.555341 6863 raft_consensus.cc:357] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.555507 6863 raft_consensus.cc:738] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.555783 6863 consensus_queue.cc:260] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.556411 6863 ts_tablet_manager.cc:1428] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250622 13:54:16.566989 6863 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Bootstrap starting.
I20250622 13:54:16.569764 413 ts_tablet_manager.cc:610] Registered 4 tablets
I20250622 13:54:16.569837 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.021s user 0.018s sys 0.003s
I20250622 13:54:16.570478 6863 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:16.570775 6863 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Bootstrap complete.
I20250622 13:54:16.571015 6863 ts_tablet_manager.cc:1397] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:54:16.571686 6863 raft_consensus.cc:357] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:16.571890 6863 raft_consensus.cc:738] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.572227 6863 consensus_queue.cc:260] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:16.572829 6863 ts_tablet_manager.cc:1428] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250622 13:54:16.573078 6863 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Bootstrap starting.
I20250622 13:54:16.577234 6863 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:16.577572 6863 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Bootstrap complete.
I20250622 13:54:16.577800 6863 ts_tablet_manager.cc:1397] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250622 13:54:16.578403 6863 raft_consensus.cc:357] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.578536 6863 raft_consensus.cc:738] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.578768 6863 consensus_queue.cc:260] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.579135 6863 ts_tablet_manager.cc:1428] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250622 13:54:16.579348 6863 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Bootstrap starting.
I20250622 13:54:16.586380 6863 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:16.586750 6863 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Bootstrap complete.
I20250622 13:54:16.587021 6863 ts_tablet_manager.cc:1397] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20250622 13:54:16.587603 6863 raft_consensus.cc:357] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.587744 6863 raft_consensus.cc:738] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.588014 6863 consensus_queue.cc:260] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.588466 6863 ts_tablet_manager.cc:1428] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250622 13:54:16.589898 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:41775
I20250622 13:54:16.589920 6924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:41775 every 8 connection(s)
I20250622 13:54:16.591125 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:16.593811 6929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:16.593961 6928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:16.598546 6925 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34635
I20250622 13:54:16.598709 6925 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:16.599190 6925 heartbeater.cc:507] Master 127.0.103.126:34635 requested a full tablet report, sending...
I20250622 13:54:16.600909 6350 ts_manager.cc:194] Re-registered known tserver with Master: 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
W20250622 13:54:16.601614 6931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:16.601647 413 server_base.cc:1048] running on GCE node
I20250622 13:54:16.602108 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:16.602166 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:16.602202 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600456602198 us; error 0 us; skew 500 ppm
I20250622 13:54:16.602340 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:16.603281 6350 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45686
I20250622 13:54:16.603930 413 webserver.cc:469] Webserver started at http://127.0.103.67:37245/ using document root <none> and password file <none>
I20250622 13:54:16.604123 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:16.604176 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250622 13:54:16.605841 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:54:16.607203 6936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:16.607535 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:16.607617 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "9f774500db5f4d6f912457b674f9fcb7"
format_stamp: "Formatted at 2025-06-22 13:54:09 on dist-test-slave-tmv0"
I20250622 13:54:16.607704 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:16.617761 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:16.618242 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:16.618741 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:16.627251 413 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250622 13:54:16.627351 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20250622 13:54:16.629190 6943 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap starting.
I20250622 13:54:16.632820 6943 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:16.633101 6943 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap complete.
I20250622 13:54:16.633302 6943 ts_tablet_manager.cc:1397] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:54:16.633682 413 ts_tablet_manager.cc:610] Registered 4 tablets
I20250622 13:54:16.633740 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.006s sys 0.000s
I20250622 13:54:16.633908 6943 raft_consensus.cc:357] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.634145 6943 raft_consensus.cc:738] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.634435 6943 consensus_queue.cc:260] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.635183 6943 ts_tablet_manager.cc:1428] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250622 13:54:16.635406 6943 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap starting.
I20250622 13:54:16.639667 6943 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:16.639973 6943 tablet_bootstrap.cc:492] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap complete.
I20250622 13:54:16.640234 6943 ts_tablet_manager.cc:1397] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250622 13:54:16.640939 6943 raft_consensus.cc:357] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:16.641219 6943 raft_consensus.cc:738] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.641557 6943 consensus_queue.cc:260] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:16.642117 6943 ts_tablet_manager.cc:1428] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:16.642369 6943 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap starting.
I20250622 13:54:16.648536 6943 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250622 13:54:16.649145 6943 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap complete.
I20250622 13:54:16.649463 6943 ts_tablet_manager.cc:1397] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20250622 13:54:16.650286 6943 raft_consensus.cc:357] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.650470 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:37405
I20250622 13:54:16.650516 7003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:37405 every 8 connection(s)
I20250622 13:54:16.650604 6943 raft_consensus.cc:738] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.650821 6943 consensus_queue.cc:260] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.651315 6943 ts_tablet_manager.cc:1428] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250622 13:54:16.651643 6943 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap starting.
I20250622 13:54:16.655992 6943 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:16.656404 6943 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Bootstrap complete.
I20250622 13:54:16.656721 6943 ts_tablet_manager.cc:1397] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20250622 13:54:16.657429 6943 raft_consensus.cc:357] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.657626 6943 raft_consensus.cc:738] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f774500db5f4d6f912457b674f9fcb7, State: Initialized, Role: FOLLOWER
I20250622 13:54:16.657873 6943 consensus_queue.cc:260] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:16.658305 6943 ts_tablet_manager.cc:1428] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:16.658434 7004 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34635
I20250622 13:54:16.658560 7004 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:16.658860 7004 heartbeater.cc:507] Master 127.0.103.126:34635 requested a full tablet report, sending...
I20250622 13:54:16.660028 6350 ts_manager.cc:194] Re-registered known tserver with Master: 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:16.661703 6350 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45692
I20250622 13:54:16.662756 7004 heartbeater.cc:499] Master 127.0.103.126:34635 was elected leader, sending a full tablet report...
W20250622 13:54:17.479192 6399 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica b6a4d13e04284fd883752fd0449a6f28 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } }
W20250622 13:54:17.480623 6399 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1414acbf4f0446ab8a76fac5307f41fd, TS 91525564458a4fa8acba6cbef9652697 -> TS 5636f6d75d3546248fbe04dd72546026 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
W20250622 13:54:17.480705 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1414acbf4f0446ab8a76fac5307f41fd, TS 91525564458a4fa8acba6cbef9652697 -> TS 5636f6d75d3546248fbe04dd72546026 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
W20250622 13:54:17.486869 6399 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9a99088323b24b71ac515b13676963a8, TS b6a4d13e04284fd883752fd0449a6f28 -> TS 5636f6d75d3546248fbe04dd72546026 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
W20250622 13:54:17.486965 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9a99088323b24b71ac515b13676963a8, TS b6a4d13e04284fd883752fd0449a6f28 -> TS 5636f6d75d3546248fbe04dd72546026 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
W20250622 13:54:17.488274 6399 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet acec09482db244de9e566ebc4e0ef93c, TS 9f774500db5f4d6f912457b674f9fcb7 -> TS 5636f6d75d3546248fbe04dd72546026 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
W20250622 13:54:17.488358 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet acec09482db244de9e566ebc4e0ef93c, TS 9f774500db5f4d6f912457b674f9fcb7 -> TS 5636f6d75d3546248fbe04dd72546026 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:17.531186 6844 heartbeater.cc:499] Master 127.0.103.126:34635 was elected leader, sending a full tablet report...
I20250622 13:54:17.604641 6925 heartbeater.cc:499] Master 127.0.103.126:34635 was elected leader, sending a full tablet report...
I20250622 13:54:17.836325 7011 raft_consensus.cc:491] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:17.836501 7011 raft_consensus.cc:513] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:17.837216 7011 leader_election.cc:290] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:17.843113 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a99088323b24b71ac515b13676963a8" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:17.843186 6899 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a99088323b24b71ac515b13676963a8" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697" is_pre_election: true
I20250622 13:54:17.843422 6899 raft_consensus.cc:2466] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:17.843449 6978 raft_consensus.cc:2408] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6a4d13e04284fd883752fd0449a6f28 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250622 13:54:17.843863 6777 leader_election.cc:304] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters: 9f774500db5f4d6f912457b674f9fcb7
I20250622 13:54:17.844134 7011 raft_consensus.cc:2802] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:54:17.844213 7011 raft_consensus.cc:491] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:17.844293 7011 raft_consensus.cc:3058] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.846135 7011 raft_consensus.cc:513] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:17.846611 7011 leader_election.cc:290] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 election: Requested vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:17.847060 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a99088323b24b71ac515b13676963a8" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:17.847060 6899 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a99088323b24b71ac515b13676963a8" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697"
I20250622 13:54:17.847270 6978 raft_consensus.cc:3058] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.847287 6899 raft_consensus.cc:3058] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.850056 6899 raft_consensus.cc:2466] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 2.
I20250622 13:54:17.850137 6978 raft_consensus.cc:2408] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b6a4d13e04284fd883752fd0449a6f28 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250622 13:54:17.850448 6777 leader_election.cc:304] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters: 9f774500db5f4d6f912457b674f9fcb7
I20250622 13:54:17.850703 7011 raft_consensus.cc:2802] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:54:17.851099 7011 raft_consensus.cc:695] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Becoming Leader. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Running, Role: LEADER
I20250622 13:54:17.851346 7011 consensus_queue.cc:237] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:17.853626 6350 catalog_manager.cc:5582] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: term changed from 1 to 2, leader changed from 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67) to b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:17.854972 7018 raft_consensus.cc:491] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:17.855134 7018 raft_consensus.cc:513] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:17.855857 7018 leader_election.cc:290] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:17.862977 6818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a4d13e04284fd883752fd0449a6f28" is_pre_election: true
I20250622 13:54:17.863060 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:17.863212 6818 raft_consensus.cc:2408] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91525564458a4fa8acba6cbef9652697 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250622 13:54:17.863246 6978 raft_consensus.cc:2466] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91525564458a4fa8acba6cbef9652697 in term 1.
I20250622 13:54:17.863610 6860 leader_election.cc:304] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, 9f774500db5f4d6f912457b674f9fcb7; no voters:
I20250622 13:54:17.863879 7018 raft_consensus.cc:2802] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:54:17.863962 7018 raft_consensus.cc:491] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:17.864029 7018 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.866101 7018 raft_consensus.cc:513] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:17.866622 7018 leader_election.cc:290] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 2 election: Requested vote from peers b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:17.867048 6818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a4d13e04284fd883752fd0449a6f28"
I20250622 13:54:17.867048 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:17.867209 6818 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.867198 6978 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.869040 6978 raft_consensus.cc:2466] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91525564458a4fa8acba6cbef9652697 in term 2.
I20250622 13:54:17.869275 6818 raft_consensus.cc:2408] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 91525564458a4fa8acba6cbef9652697 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250622 13:54:17.869414 6860 leader_election.cc:304] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, 9f774500db5f4d6f912457b674f9fcb7; no voters:
I20250622 13:54:17.869673 7018 raft_consensus.cc:2802] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:54:17.870064 7018 raft_consensus.cc:695] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 2 LEADER]: Becoming Leader. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Running, Role: LEADER
I20250622 13:54:17.870321 7018 consensus_queue.cc:237] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:17.872736 6350 catalog_manager.cc:5582] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 reported cstate change: term changed from 1 to 2, leader changed from b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65) to 91525564458a4fa8acba6cbef9652697 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "91525564458a4fa8acba6cbef9652697" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: HEALTHY } } }
I20250622 13:54:17.989393 7011 raft_consensus.cc:491] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:17.989588 7011 raft_consensus.cc:513] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:17.990242 7011 leader_election.cc:290] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:17.990835 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1414acbf4f0446ab8a76fac5307f41fd" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:17.990835 6899 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1414acbf4f0446ab8a76fac5307f41fd" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697" is_pre_election: true
I20250622 13:54:17.991091 6978 raft_consensus.cc:2466] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:17.991092 6899 raft_consensus.cc:2466] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:17.991549 6777 leader_election.cc:304] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:17.991782 7011 raft_consensus.cc:2802] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:54:17.991906 7011 raft_consensus.cc:491] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:17.991981 7011 raft_consensus.cc:3058] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.994256 7011 raft_consensus.cc:513] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:17.994779 7011 leader_election.cc:290] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 election: Requested vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:17.995337 6899 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1414acbf4f0446ab8a76fac5307f41fd" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697"
I20250622 13:54:17.995321 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1414acbf4f0446ab8a76fac5307f41fd" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:17.995535 6899 raft_consensus.cc:3058] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.995539 6978 raft_consensus.cc:3058] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.996008 7011 raft_consensus.cc:491] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:17.996145 7011 raft_consensus.cc:513] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:17.996763 7011 leader_election.cc:290] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:17.997202 6898 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697" is_pre_election: true
I20250622 13:54:17.997375 6898 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:17.997802 6977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:17.997925 6777 leader_election.cc:304] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:17.997992 6977 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 1.
I20250622 13:54:17.998172 7011 raft_consensus.cc:2802] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:54:17.998241 7011 raft_consensus.cc:491] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:17.998317 7011 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:17.998376 6978 raft_consensus.cc:2466] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 2.
I20250622 13:54:17.998438 6899 raft_consensus.cc:2466] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 2.
I20250622 13:54:17.998736 6777 leader_election.cc:304] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:17.998960 7015 raft_consensus.cc:2802] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:54:17.999392 7015 raft_consensus.cc:695] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Becoming Leader. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Running, Role: LEADER
I20250622 13:54:17.999655 7015 consensus_queue.cc:237] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:18.000725 7011 raft_consensus.cc:513] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:18.001319 7011 leader_election.cc:290] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 election: Requested vote from peers 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:18.001750 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:18.001817 6899 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697"
I20250622 13:54:18.001896 6978 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:18.002013 6899 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:18.002291 6350 catalog_manager.cc:5582] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:18.004282 6899 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 2.
I20250622 13:54:18.004348 6978 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 2.
I20250622 13:54:18.004696 6777 leader_election.cc:304] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, 9f774500db5f4d6f912457b674f9fcb7, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:18.004946 7011 raft_consensus.cc:2802] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:54:18.005116 7011 raft_consensus.cc:695] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Becoming Leader. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Running, Role: LEADER
I20250622 13:54:18.005430 7011 consensus_queue.cc:237] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } }
I20250622 13:54:18.007664 6350 catalog_manager.cc:5582] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:18.008158 7011 raft_consensus.cc:491] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:18.008257 7011 raft_consensus.cc:513] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:18.008702 7011 leader_election.cc:290] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405), 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
I20250622 13:54:18.009141 6899 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697" is_pre_election: true
I20250622 13:54:18.009171 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7" is_pre_election: true
I20250622 13:54:18.009317 6978 raft_consensus.cc:2466] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 2.
I20250622 13:54:18.009655 6779 leader_election.cc:304] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f774500db5f4d6f912457b674f9fcb7, b6a4d13e04284fd883752fd0449a6f28; no voters:
I20250622 13:54:18.009982 7011 raft_consensus.cc:2802] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250622 13:54:18.010092 7011 raft_consensus.cc:491] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:18.010229 7011 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:54:18.012230 7011 raft_consensus.cc:513] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:18.012701 7011 leader_election.cc:290] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 3 election: Requested vote from peers 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405), 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
I20250622 13:54:18.013162 6899 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91525564458a4fa8acba6cbef9652697"
I20250622 13:54:18.013163 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9994feeb6d842f694ef146799ba1e38" candidate_uuid: "b6a4d13e04284fd883752fd0449a6f28" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:18.013419 6978 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:54:18.015309 6978 raft_consensus.cc:2466] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a4d13e04284fd883752fd0449a6f28 in term 3.
I20250622 13:54:18.015655 6779 leader_election.cc:304] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f774500db5f4d6f912457b674f9fcb7, b6a4d13e04284fd883752fd0449a6f28; no voters: 91525564458a4fa8acba6cbef9652697
I20250622 13:54:18.015960 7011 raft_consensus.cc:2802] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:54:18.016121 7011 raft_consensus.cc:695] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 3 LEADER]: Becoming Leader. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Running, Role: LEADER
I20250622 13:54:18.016341 7011 consensus_queue.cc:237] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:18.018507 6350 catalog_manager.cc:5582] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: term changed from 2 to 3, leader changed from 91525564458a4fa8acba6cbef9652697 (127.0.103.66) to b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65). New cstate: current_term: 3 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:18.234719 6978 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250622 13:54:18.234874 6978 pending_rounds.cc:85] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Aborting all ops after (but not including) 0
I20250622 13:54:18.234948 6978 pending_rounds.cc:107] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250622 13:54:18.235029 6978 raft_consensus.cc:2887] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 13:54:18.235746 7011 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.239746 6899 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250622 13:54:18.240337 7026 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.276108 6978 raft_consensus.cc:1238] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Rejecting Update request from peer 91525564458a4fa8acba6cbef9652697 for earlier term 2. Current term is 3. Ops: []
I20250622 13:54:18.276758 7018 consensus_queue.cc:1046] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.277051 7024 raft_consensus.cc:3053] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:54:18.277155 7024 raft_consensus.cc:738] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Running, Role: LEADER
I20250622 13:54:18.277463 7024 consensus_queue.cc:260] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } }
I20250622 13:54:18.277787 7024 raft_consensus.cc:3058] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:54:18.400585 6899 raft_consensus.cc:1273] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:54:18.401208 7026 consensus_queue.cc:1035] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.405730 6978 raft_consensus.cc:1273] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:54:18.406235 7015 consensus_queue.cc:1035] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.419358 6978 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:54:18.419945 7015 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.423768 6899 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:54:18.424388 7015 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.492399 6818 consensus_queue.cc:237] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:18.494589 6899 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:54:18.495349 7011 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:18.495834 6978 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:54:18.496376 7011 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:18.497718 7026 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.498571 6899 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.499465 6978 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
W20250622 13:54:18.503674 6776 consensus_peers.cc:487] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 -> Peer 5636f6d75d3546248fbe04dd72546026 (127.0.103.68:35959): Couldn't send request to peer 5636f6d75d3546248fbe04dd72546026. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a99088323b24b71ac515b13676963a8. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:18.504050 6818 consensus_queue.cc:237] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:18.504544 6350 catalog_manager.cc:5582] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: config changed from index -1 to 2, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:18.506604 6899 raft_consensus.cc:1273] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250622 13:54:18.506737 6776 consensus_peers.cc:487] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 -> Peer 5636f6d75d3546248fbe04dd72546026 (127.0.103.68:35959): Couldn't send request to peer 5636f6d75d3546248fbe04dd72546026. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acec09482db244de9e566ebc4e0ef93c. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:18.507141 7026 consensus_queue.cc:1035] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:18.509680 6978 raft_consensus.cc:1273] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:18.509593 7011 raft_consensus.cc:2953] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.510237 7042 consensus_queue.cc:1035] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:18.512534 6978 raft_consensus.cc:2953] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.513844 6899 raft_consensus.cc:2953] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.517477 6818 consensus_queue.cc:237] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:18.517928 6351 catalog_manager.cc:5582] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: config changed from index -1 to 3, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250622 13:54:18.519657 6776 consensus_peers.cc:487] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 -> Peer 5636f6d75d3546248fbe04dd72546026 (127.0.103.68:35959): Couldn't send request to peer 5636f6d75d3546248fbe04dd72546026. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1414acbf4f0446ab8a76fac5307f41fd. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:18.520056 6899 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:18.520197 6978 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:18.520651 7026 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:18.520890 7042 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:18.523074 7015 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.523480 6899 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.527813 6351 catalog_manager.cc:5582] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 reported cstate change: config changed from index -1 to 3, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.530515 6978 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } } }
I20250622 13:54:18.532795 6818 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c"
dest_uuid: "b6a4d13e04284fd883752fd0449a6f28"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54630
I20250622 13:54:18.532999 6818 raft_consensus.cc:604] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Received request to transfer leadership
I20250622 13:54:18.538118 6818 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c"
dest_uuid: "b6a4d13e04284fd883752fd0449a6f28"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54630
I20250622 13:54:18.538300 6818 raft_consensus.cc:604] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Received request to transfer leadership
W20250622 13:54:18.538607 6395 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for acec09482db244de9e566ebc4e0ef93c already in progress [suppressed 14 similar messages]
I20250622 13:54:18.539628 6818 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c"
dest_uuid: "b6a4d13e04284fd883752fd0449a6f28"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54630
I20250622 13:54:18.539772 6818 raft_consensus.cc:604] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Received request to transfer leadership
W20250622 13:54:18.540237 6399 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for acec09482db244de9e566ebc4e0ef93c already in progress
W20250622 13:54:18.540354 6399 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for acec09482db244de9e566ebc4e0ef93c already in progress
I20250622 13:54:18.547807 6978 raft_consensus.cc:1273] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250622 13:54:18.548357 7015 consensus_queue.cc:1035] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.554311 6899 raft_consensus.cc:1273] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 3 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250622 13:54:18.554968 7011 consensus_queue.cc:1035] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:18.555233 7011 consensus_queue.cc:1230] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Peer 91525564458a4fa8acba6cbef9652697 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250622 13:54:18.556365 6899 pending_rounds.cc:85] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Aborting all ops after (but not including) 0
I20250622 13:54:18.556476 6899 pending_rounds.cc:107] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250622 13:54:18.556552 6899 raft_consensus.cc:2887] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250622 13:54:19.020318 7045 ts_tablet_manager.cc:927] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: Initiating tablet copy from peer b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653)
I20250622 13:54:19.021065 7011 raft_consensus.cc:991] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28: : Instructing follower 91525564458a4fa8acba6cbef9652697 to start an election
I20250622 13:54:19.021145 7045 tablet_copy_client.cc:323] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:38653
I20250622 13:54:19.021221 7011 raft_consensus.cc:1079] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Signalling peer 91525564458a4fa8acba6cbef9652697 to start an election
I20250622 13:54:19.021795 6899 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c"
dest_uuid: "91525564458a4fa8acba6cbef9652697"
from {username='slave'} at 127.0.0.1:44308
I20250622 13:54:19.022006 6899 raft_consensus.cc:491] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:54:19.022143 6899 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:54:19.024870 6899 raft_consensus.cc:513] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.025765 6899 leader_election.cc:290] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 3 election: Requested vote from peers b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653), 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405)
I20250622 13:54:19.026347 6818 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "b6a4d13e04284fd883752fd0449a6f28"
I20250622 13:54:19.026522 6818 raft_consensus.cc:3053] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:54:19.026382 6978 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acec09482db244de9e566ebc4e0ef93c" candidate_uuid: "91525564458a4fa8acba6cbef9652697" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "9f774500db5f4d6f912457b674f9fcb7"
I20250622 13:54:19.026597 6818 raft_consensus.cc:738] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b6a4d13e04284fd883752fd0449a6f28, State: Running, Role: LEADER
I20250622 13:54:19.026646 6978 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:54:19.026844 6818 consensus_queue.cc:260] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.027223 6818 raft_consensus.cc:3058] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:54:19.029583 6978 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91525564458a4fa8acba6cbef9652697 in term 3.
I20250622 13:54:19.030192 6829 tablet_copy_service.cc:140] P b6a4d13e04284fd883752fd0449a6f28: Received BeginTabletCopySession request for tablet 1414acbf4f0446ab8a76fac5307f41fd from peer 5636f6d75d3546248fbe04dd72546026 ({username='slave'} at 127.0.0.1:54652)
I20250622 13:54:19.030321 6829 tablet_copy_service.cc:161] P b6a4d13e04284fd883752fd0449a6f28: Beginning new tablet copy session on tablet 1414acbf4f0446ab8a76fac5307f41fd from peer 5636f6d75d3546248fbe04dd72546026 at {username='slave'} at 127.0.0.1:54652: session id = 5636f6d75d3546248fbe04dd72546026-1414acbf4f0446ab8a76fac5307f41fd
I20250622 13:54:19.030426 6860 leader_election.cc:304] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91525564458a4fa8acba6cbef9652697, 9f774500db5f4d6f912457b674f9fcb7; no voters:
I20250622 13:54:19.030694 6818 raft_consensus.cc:2466] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91525564458a4fa8acba6cbef9652697 in term 3.
I20250622 13:54:19.031509 7049 ts_tablet_manager.cc:927] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: Initiating tablet copy from peer b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65:38653)
I20250622 13:54:19.032048 7049 tablet_copy_client.cc:323] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:38653
I20250622 13:54:19.032591 6828 tablet_copy_service.cc:140] P b6a4d13e04284fd883752fd0449a6f28: Received BeginTabletCopySession request for tablet 9a99088323b24b71ac515b13676963a8 from peer 5636f6d75d3546248fbe04dd72546026 ({username='slave'} at 127.0.0.1:54652)
I20250622 13:54:19.032683 6828 tablet_copy_service.cc:161] P b6a4d13e04284fd883752fd0449a6f28: Beginning new tablet copy session on tablet 9a99088323b24b71ac515b13676963a8 from peer 5636f6d75d3546248fbe04dd72546026 at {username='slave'} at 127.0.0.1:54652: session id = 5636f6d75d3546248fbe04dd72546026-9a99088323b24b71ac515b13676963a8
I20250622 13:54:19.033180 6829 tablet_copy_source_session.cc:215] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:19.033489 7018 raft_consensus.cc:2802] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:54:19.033601 7018 raft_consensus.cc:695] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 3 LEADER]: Becoming Leader. State: Replica: 91525564458a4fa8acba6cbef9652697, State: Running, Role: LEADER
I20250622 13:54:19.033835 7018 consensus_queue.cc:237] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.034212 7045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1414acbf4f0446ab8a76fac5307f41fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:19.035158 6828 tablet_copy_source_session.cc:215] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:19.036381 7049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a99088323b24b71ac515b13676963a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:19.036857 6350 catalog_manager.cc:5582] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 reported cstate change: term changed from 2 to 3, leader changed from b6a4d13e04284fd883752fd0449a6f28 (127.0.103.65) to 91525564458a4fa8acba6cbef9652697 (127.0.103.66). New cstate: current_term: 3 leader_uuid: "91525564458a4fa8acba6cbef9652697" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:19.041378 7045 tablet_copy_client.cc:806] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:19.041739 7045 tablet_copy_client.cc:670] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:19.043579 7045 tablet_copy_client.cc:538] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:19.044411 7049 tablet_copy_client.cc:806] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:19.044634 7049 tablet_copy_client.cc:670] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:19.046221 7049 tablet_copy_client.cc:538] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:19.049315 7045 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: Bootstrap starting.
I20250622 13:54:19.050537 7049 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: Bootstrap starting.
I20250622 13:54:19.057380 7049 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:19.057775 7049 tablet_bootstrap.cc:492] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: Bootstrap complete.
I20250622 13:54:19.057849 7045 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:19.058010 7049 ts_tablet_manager.cc:1397] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20250622 13:54:19.058194 7045 tablet_bootstrap.cc:492] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: Bootstrap complete.
I20250622 13:54:19.058432 7045 ts_tablet_manager.cc:1397] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: Time spent bootstrapping tablet: real 0.009s user 0.010s sys 0.000s
I20250622 13:54:19.058717 7049 raft_consensus.cc:357] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.058925 7049 raft_consensus.cc:738] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5636f6d75d3546248fbe04dd72546026, State: Initialized, Role: LEARNER
I20250622 13:54:19.059022 7045 raft_consensus.cc:357] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.059149 7049 consensus_queue.cc:260] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.059615 7045 raft_consensus.cc:738] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5636f6d75d3546248fbe04dd72546026, State: Initialized, Role: LEARNER
I20250622 13:54:19.059798 7045 consensus_queue.cc:260] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.060451 7045 ts_tablet_manager.cc:1428] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:19.061187 6828 tablet_copy_service.cc:342] P b6a4d13e04284fd883752fd0449a6f28: Request end of tablet copy session 5636f6d75d3546248fbe04dd72546026-1414acbf4f0446ab8a76fac5307f41fd received from {username='slave'} at 127.0.0.1:54652
I20250622 13:54:19.061295 6828 tablet_copy_service.cc:434] P b6a4d13e04284fd883752fd0449a6f28: ending tablet copy session 5636f6d75d3546248fbe04dd72546026-1414acbf4f0446ab8a76fac5307f41fd on tablet 1414acbf4f0446ab8a76fac5307f41fd with peer 5636f6d75d3546248fbe04dd72546026
I20250622 13:54:19.061473 7049 ts_tablet_manager.cc:1428] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:54:19.062449 6828 tablet_copy_service.cc:342] P b6a4d13e04284fd883752fd0449a6f28: Request end of tablet copy session 5636f6d75d3546248fbe04dd72546026-9a99088323b24b71ac515b13676963a8 received from {username='slave'} at 127.0.0.1:54652
I20250622 13:54:19.062556 6828 tablet_copy_service.cc:434] P b6a4d13e04284fd883752fd0449a6f28: ending tablet copy session 5636f6d75d3546248fbe04dd72546026-9a99088323b24b71ac515b13676963a8 on tablet 9a99088323b24b71ac515b13676963a8 with peer 5636f6d75d3546248fbe04dd72546026
I20250622 13:54:19.428318 6818 raft_consensus.cc:1273] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 3 FOLLOWER]: Refusing update from remote peer 91525564458a4fa8acba6cbef9652697: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250622 13:54:19.428821 7018 consensus_queue.cc:1035] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:19.433779 6977 raft_consensus.cc:1273] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Refusing update from remote peer 91525564458a4fa8acba6cbef9652697: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250622 13:54:19.434602 7018 consensus_queue.cc:1035] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250622 13:54:19.446769 6857 consensus_peers.cc:487] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 -> Peer 5636f6d75d3546248fbe04dd72546026 (127.0.103.68:35959): Couldn't send request to peer 5636f6d75d3546248fbe04dd72546026. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acec09482db244de9e566ebc4e0ef93c. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:19.550266 6723 raft_consensus.cc:1215] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250622 13:54:19.558480 6723 raft_consensus.cc:1215] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250622 13:54:19.964893 7066 ts_tablet_manager.cc:927] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: Initiating tablet copy from peer 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775)
I20250622 13:54:19.965561 7066 tablet_copy_client.cc:323] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:41775
I20250622 13:54:19.971477 6910 tablet_copy_service.cc:140] P 91525564458a4fa8acba6cbef9652697: Received BeginTabletCopySession request for tablet acec09482db244de9e566ebc4e0ef93c from peer 5636f6d75d3546248fbe04dd72546026 ({username='slave'} at 127.0.0.1:44324)
I20250622 13:54:19.971606 6910 tablet_copy_service.cc:161] P 91525564458a4fa8acba6cbef9652697: Beginning new tablet copy session on tablet acec09482db244de9e566ebc4e0ef93c from peer 5636f6d75d3546248fbe04dd72546026 at {username='slave'} at 127.0.0.1:44324: session id = 5636f6d75d3546248fbe04dd72546026-acec09482db244de9e566ebc4e0ef93c
I20250622 13:54:19.973474 6910 tablet_copy_source_session.cc:215] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:19.974663 7066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acec09482db244de9e566ebc4e0ef93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:19.980836 7066 tablet_copy_client.cc:806] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:19.981045 7066 tablet_copy_client.cc:670] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:19.982508 7066 tablet_copy_client.cc:538] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:19.985560 7066 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: Bootstrap starting.
I20250622 13:54:19.992635 7066 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:19.993045 7066 tablet_bootstrap.cc:492] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: Bootstrap complete.
I20250622 13:54:19.993287 7066 ts_tablet_manager.cc:1397] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20250622 13:54:19.994118 7066 raft_consensus.cc:357] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.994313 7066 raft_consensus.cc:738] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5636f6d75d3546248fbe04dd72546026, State: Initialized, Role: LEARNER
I20250622 13:54:19.994495 7066 consensus_queue.cc:260] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } attrs { replace: true } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: true } }
I20250622 13:54:19.995018 7066 ts_tablet_manager.cc:1428] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:54:19.995581 6910 tablet_copy_service.cc:342] P 91525564458a4fa8acba6cbef9652697: Request end of tablet copy session 5636f6d75d3546248fbe04dd72546026-acec09482db244de9e566ebc4e0ef93c received from {username='slave'} at 127.0.0.1:44324
I20250622 13:54:19.995687 6910 tablet_copy_service.cc:434] P 91525564458a4fa8acba6cbef9652697: ending tablet copy session 5636f6d75d3546248fbe04dd72546026-acec09482db244de9e566ebc4e0ef93c on tablet acec09482db244de9e566ebc4e0ef93c with peer 5636f6d75d3546248fbe04dd72546026
I20250622 13:54:20.125828 7073 raft_consensus.cc:1062] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28: attempting to promote NON_VOTER 5636f6d75d3546248fbe04dd72546026 to VOTER
I20250622 13:54:20.126665 7073 consensus_queue.cc:237] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } }
I20250622 13:54:20.128957 6723 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 LEARNER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:20.129060 6977 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:20.129472 6899 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:20.129601 7042 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:20.130069 7073 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:20.130296 7042 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:20.133056 7073 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.133687 6977 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.137962 6899 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.138967 6350 catalog_manager.cc:5582] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: config changed from index 3 to 4, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:20.141892 6723 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.146415 7075 raft_consensus.cc:1062] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28: attempting to promote NON_VOTER 5636f6d75d3546248fbe04dd72546026 to VOTER
I20250622 13:54:20.147233 7075 consensus_queue.cc:237] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } }
I20250622 13:54:20.149778 6723 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 LEARNER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:20.150444 6977 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:20.150573 6899 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:20.150344 7075 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:20.151028 7042 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:20.151298 7075 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:20.153805 7042 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.154465 6977 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.157943 6723 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.160837 6899 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.166390 6351 catalog_manager.cc:5582] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 reported cstate change: config changed from index 2 to 3, 5636f6d75d3546248fbe04dd72546026 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:54:20.170715 6817 consensus_queue.cc:237] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } }
I20250622 13:54:20.171104 6816 consensus_queue.cc:237] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } }
I20250622 13:54:20.172359 6977 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250622 13:54:20.172680 6723 raft_consensus.cc:1273] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250622 13:54:20.172755 6722 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:20.173177 7073 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:20.173477 7042 consensus_queue.cc:1035] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:20.173590 7075 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:20.174126 6899 raft_consensus.cc:1273] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Refusing update from remote peer b6a4d13e04284fd883752fd0449a6f28: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:20.176014 7042 consensus_queue.cc:1035] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:20.176659 7042 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 91525564458a4fa8acba6cbef9652697 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.177699 7075 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.180500 6395 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1414acbf4f0446ab8a76fac5307f41fd with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:20.181201 6723 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 91525564458a4fa8acba6cbef9652697 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.181855 6395 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9a99088323b24b71ac515b13676963a8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:20.183300 6899 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.185518 6349 catalog_manager.cc:5582] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 reported cstate change: config changed from index 4 to 5, VOTER 91525564458a4fa8acba6cbef9652697 (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.190819 6977 raft_consensus.cc:2953] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 91525564458a4fa8acba6cbef9652697 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "9f774500db5f4d6f912457b674f9fcb7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 37405 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.191430 6723 raft_consensus.cc:2953] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.192922 6348 catalog_manager.cc:5582] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 reported cstate change: config changed from index 3 to 4, VOTER 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "b6a4d13e04284fd883752fd0449a6f28" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b6a4d13e04284fd883752fd0449a6f28" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38653 } } peers { permanent_uuid: "91525564458a4fa8acba6cbef9652697" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41775 } } peers { permanent_uuid: "5636f6d75d3546248fbe04dd72546026" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 35959 } attrs { promote: false } } }
I20250622 13:54:20.201064 6888 tablet_service.cc:1515] Processing DeleteTablet for tablet 1414acbf4f0446ab8a76fac5307f41fd with delete_type TABLET_DATA_TOMBSTONED (TS 91525564458a4fa8acba6cbef9652697 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44338
I20250622 13:54:20.201776 7094 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.202016 7094 raft_consensus.cc:2241] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.202304 7094 raft_consensus.cc:2270] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.203049 7094 ts_tablet_manager.cc:1905] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:54:20.207003 6968 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a99088323b24b71ac515b13676963a8 with delete_type TABLET_DATA_TOMBSTONED (TS 9f774500db5f4d6f912457b674f9fcb7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56954
I20250622 13:54:20.207595 7095 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.207746 7095 raft_consensus.cc:2241] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.207954 7095 raft_consensus.cc:2270] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.208611 7094 ts_tablet_manager.cc:1918] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250622 13:54:20.208699 7094 log.cc:1199] T 1414acbf4f0446ab8a76fac5307f41fd P 91525564458a4fa8acba6cbef9652697: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-1-root/wals/1414acbf4f0446ab8a76fac5307f41fd
I20250622 13:54:20.208756 7095 ts_tablet_manager.cc:1905] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:54:20.209347 6336 catalog_manager.cc:4928] TS 91525564458a4fa8acba6cbef9652697 (127.0.103.66:41775): tablet 1414acbf4f0446ab8a76fac5307f41fd (table test-workload [id=2d7ecd54faad4dfdac3025b990a83895]) successfully deleted
I20250622 13:54:20.212868 7095 ts_tablet_manager.cc:1918] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20250622 13:54:20.212962 7095 log.cc:1199] T 9a99088323b24b71ac515b13676963a8 P 9f774500db5f4d6f912457b674f9fcb7: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750600281110842-413-0/minicluster-data/ts-2-root/wals/9a99088323b24b71ac515b13676963a8
I20250622 13:54:20.213600 6338 catalog_manager.cc:4928] TS 9f774500db5f4d6f912457b674f9fcb7 (127.0.103.67:37405): tablet 9a99088323b24b71ac515b13676963a8 (table test-workload [id=2d7ecd54faad4dfdac3025b990a83895]) successfully deleted
I20250622 13:54:20.532051 6723 raft_consensus.cc:1215] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026 [term 3 LEARNER]: Deduplicated request from leader. Original: 2.3->[3.4-3.4] Dedup: 3.4->[]
I20250622 13:54:20.678339 413 tablet_server.cc:178] TabletServer@127.0.103.65:38653 shutting down...
I20250622 13:54:20.686358 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:20.686666 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.686834 413 raft_consensus.cc:2241] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:54:20.687147 413 raft_consensus.cc:2270] T 1414acbf4f0446ab8a76fac5307f41fd P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.687878 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.688017 413 raft_consensus.cc:2241] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:54:20.688244 413 raft_consensus.cc:2270] T 9a99088323b24b71ac515b13676963a8 P b6a4d13e04284fd883752fd0449a6f28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.688815 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.688941 413 raft_consensus.cc:2241] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.689096 413 raft_consensus.cc:2270] T acec09482db244de9e566ebc4e0ef93c P b6a4d13e04284fd883752fd0449a6f28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.689543 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.689661 413 raft_consensus.cc:2241] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 3 LEADER]: Raft consensus shutting down.
I20250622 13:54:20.689869 413 raft_consensus.cc:2270] T c9994feeb6d842f694ef146799ba1e38 P b6a4d13e04284fd883752fd0449a6f28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.694630 413 tablet_server.cc:195] TabletServer@127.0.103.65:38653 shutdown complete.
I20250622 13:54:20.698355 413 tablet_server.cc:178] TabletServer@127.0.103.66:41775 shutting down...
I20250622 13:54:20.707307 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:20.707695 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.707849 413 raft_consensus.cc:2241] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.708041 413 raft_consensus.cc:2270] T 9a99088323b24b71ac515b13676963a8 P 91525564458a4fa8acba6cbef9652697 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.708899 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.709031 413 raft_consensus.cc:2241] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 3 LEADER]: Raft consensus shutting down.
I20250622 13:54:20.709389 413 raft_consensus.cc:2270] T acec09482db244de9e566ebc4e0ef93c P 91525564458a4fa8acba6cbef9652697 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.709856 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.710001 413 raft_consensus.cc:2241] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.710147 413 raft_consensus.cc:2270] T c9994feeb6d842f694ef146799ba1e38 P 91525564458a4fa8acba6cbef9652697 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.714681 413 tablet_server.cc:195] TabletServer@127.0.103.66:41775 shutdown complete.
I20250622 13:54:20.718101 413 tablet_server.cc:178] TabletServer@127.0.103.67:37405 shutting down...
I20250622 13:54:20.725977 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:20.726207 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.726394 413 raft_consensus.cc:2241] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.726634 413 raft_consensus.cc:2270] T 1414acbf4f0446ab8a76fac5307f41fd P 9f774500db5f4d6f912457b674f9fcb7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.727397 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.727506 413 raft_consensus.cc:2241] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.727641 413 raft_consensus.cc:2270] T acec09482db244de9e566ebc4e0ef93c P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.728008 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:20.728096 413 raft_consensus.cc:2241] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.728189 413 raft_consensus.cc:2270] T c9994feeb6d842f694ef146799ba1e38 P 9f774500db5f4d6f912457b674f9fcb7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.742377 413 tablet_server.cc:195] TabletServer@127.0.103.67:37405 shutdown complete.
I20250622 13:54:20.745851 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250622 13:54:20.759652 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:20.759892 413 tablet_replica.cc:331] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026: stopping tablet replica
I20250622 13:54:20.760058 413 raft_consensus.cc:2241] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026 [term 3 LEARNER]: Raft consensus shutting down.
I20250622 13:54:20.760218 413 raft_consensus.cc:2270] T acec09482db244de9e566ebc4e0ef93c P 5636f6d75d3546248fbe04dd72546026 [term 3 LEARNER]: Raft consensus is shut down!
I20250622 13:54:20.760782 413 tablet_replica.cc:331] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026: stopping tablet replica
I20250622 13:54:20.760913 413 raft_consensus.cc:2241] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.761132 413 raft_consensus.cc:2270] T 1414acbf4f0446ab8a76fac5307f41fd P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.761781 413 tablet_replica.cc:331] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026: stopping tablet replica
I20250622 13:54:20.761871 413 raft_consensus.cc:2241] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:20.762029 413 raft_consensus.cc:2270] T 9a99088323b24b71ac515b13676963a8 P 5636f6d75d3546248fbe04dd72546026 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.776058 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250622 13:54:20.779364 413 master.cc:561] Master@127.0.103.126:34635 shutting down...
I20250622 13:54:20.786504 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:20.786705 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:20.786792 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 71a871796b614b7e8b4ad5231cdb9747: stopping tablet replica
I20250622 13:54:20.800310 413 master.cc:583] Master@127.0.103.126:34635 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11470 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20250622 13:54:20.813640 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:44101
I20250622 13:54:20.814119 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:20.816383 7098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:20.816457 7099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:20.816598 7101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:20.816965 413 server_base.cc:1048] running on GCE node
I20250622 13:54:20.817226 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:20.817274 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:20.817317 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600460817313 us; error 0 us; skew 500 ppm
I20250622 13:54:20.817480 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:20.818641 413 webserver.cc:469] Webserver started at http://127.0.103.126:44141/ using document root <none> and password file <none>
I20250622 13:54:20.818866 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:20.818933 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:20.819042 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:20.819715 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/master-0-root/instance:
uuid: "cf3d5833c39d490d82e5b74e1dba281c"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:20.822327 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:54:20.823761 7106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:20.824095 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:20.824183 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/master-0-root
uuid: "cf3d5833c39d490d82e5b74e1dba281c"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:20.824316 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:20.836326 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:20.836820 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:20.849774 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:44101
I20250622 13:54:20.849817 7157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:44101 every 8 connection(s)
I20250622 13:54:20.851080 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:20.855252 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: Bootstrap starting.
I20250622 13:54:20.856369 7158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:20.857803 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: No bootstrap required, opened a new log
I20250622 13:54:20.858361 7158 raft_consensus.cc:357] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3d5833c39d490d82e5b74e1dba281c" member_type: VOTER }
I20250622 13:54:20.858482 7158 raft_consensus.cc:383] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:20.858516 7158 raft_consensus.cc:738] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3d5833c39d490d82e5b74e1dba281c, State: Initialized, Role: FOLLOWER
I20250622 13:54:20.858691 7158 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3d5833c39d490d82e5b74e1dba281c" member_type: VOTER }
I20250622 13:54:20.858806 7158 raft_consensus.cc:397] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:54:20.858842 7158 raft_consensus.cc:491] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:54:20.858891 7158 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:20.860674 7158 raft_consensus.cc:513] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3d5833c39d490d82e5b74e1dba281c" member_type: VOTER }
I20250622 13:54:20.860827 7158 leader_election.cc:304] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf3d5833c39d490d82e5b74e1dba281c; no voters:
I20250622 13:54:20.861150 7158 leader_election.cc:290] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:54:20.861284 7161 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:20.861778 7161 raft_consensus.cc:695] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 1 LEADER]: Becoming Leader. State: Replica: cf3d5833c39d490d82e5b74e1dba281c, State: Running, Role: LEADER
I20250622 13:54:20.861986 7161 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3d5833c39d490d82e5b74e1dba281c" member_type: VOTER }
I20250622 13:54:20.862066 7158 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:54:20.862762 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf3d5833c39d490d82e5b74e1dba281c. Latest consensus state: current_term: 1 leader_uuid: "cf3d5833c39d490d82e5b74e1dba281c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3d5833c39d490d82e5b74e1dba281c" member_type: VOTER } }
I20250622 13:54:20.862895 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [sys.catalog]: This master's current role is: LEADER
I20250622 13:54:20.862811 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf3d5833c39d490d82e5b74e1dba281c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3d5833c39d490d82e5b74e1dba281c" member_type: VOTER } }
I20250622 13:54:20.863144 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [sys.catalog]: This master's current role is: LEADER
I20250622 13:54:20.865756 7166 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:54:20.867048 7166 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:54:20.868554 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250622 13:54:20.869786 7166 catalog_manager.cc:1349] Generated new cluster ID: 3683a3d049b9478b86220ab051cbefe2
I20250622 13:54:20.869863 7166 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:54:20.877462 7166 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:54:20.878222 7166 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:54:20.887707 7166 catalog_manager.cc:5955] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: Generated new TSK 0
I20250622 13:54:20.887969 7166 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:54:20.901474 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:20.903934 7179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:20.904513 7182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:20.904448 7180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:20.904882 413 server_base.cc:1048] running on GCE node
I20250622 13:54:20.905153 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:20.905226 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:20.905261 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600460905257 us; error 0 us; skew 500 ppm
I20250622 13:54:20.905464 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:20.906687 413 webserver.cc:469] Webserver started at http://127.0.103.65:34253/ using document root <none> and password file <none>
I20250622 13:54:20.906898 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:20.906967 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:20.907096 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:20.907752 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root/instance:
uuid: "cc8abf945e6841afb6734436277e357a"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:20.910336 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:54:20.911846 7187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:20.912182 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250622 13:54:20.912300 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "cc8abf945e6841afb6734436277e357a"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:20.912413 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:20.921631 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:20.922127 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:20.922655 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:20.923516 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:54:20.923571 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:20.923650 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:54:20.923686 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:20.936053 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41121
I20250622 13:54:20.936075 7249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41121 every 8 connection(s)
I20250622 13:54:20.937650 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:20.940441 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:20.940770 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:20.941278 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:20.942642 413 server_base.cc:1048] running on GCE node
I20250622 13:54:20.942932 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:20.943004 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:20.943055 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600460943047 us; error 0 us; skew 500 ppm
I20250622 13:54:20.943212 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:20.944453 7250 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44101
I20250622 13:54:20.944583 7250 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:20.944592 413 webserver.cc:469] Webserver started at http://127.0.103.66:38947/ using document root <none> and password file <none>
I20250622 13:54:20.944808 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:20.944862 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:20.944924 7250 heartbeater.cc:507] Master 127.0.103.126:44101 requested a full tablet report, sending...
I20250622 13:54:20.944967 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:20.945590 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root/instance:
uuid: "c0f475d5335b4087a1319af1128b2071"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:20.945772 7123 ts_manager.cc:194] Registered new tserver with Master: cc8abf945e6841afb6734436277e357a (127.0.103.65:41121)
I20250622 13:54:20.946732 7123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34154
I20250622 13:54:20.948036 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:20.949546 7262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:20.949877 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250622 13:54:20.949970 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root
uuid: "c0f475d5335b4087a1319af1128b2071"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:20.950085 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:20.956115 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:20.956569 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:20.957062 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:20.957901 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:54:20.957957 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:20.958058 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:54:20.958101 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:20.970455 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:45387
I20250622 13:54:20.970480 7324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:45387 every 8 connection(s)
I20250622 13:54:20.971920 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:20.974824 7328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:20.974824 7329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:20.978922 7331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:20.979460 7325 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44101
I20250622 13:54:20.979660 413 server_base.cc:1048] running on GCE node
I20250622 13:54:20.979791 7325 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:20.979879 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:20.979945 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:20.979988 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600460979984 us; error 0 us; skew 500 ppm
I20250622 13:54:20.980120 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:20.980268 7325 heartbeater.cc:507] Master 127.0.103.126:44101 requested a full tablet report, sending...
I20250622 13:54:20.981436 7123 ts_manager.cc:194] Registered new tserver with Master: c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387)
I20250622 13:54:20.981665 413 webserver.cc:469] Webserver started at http://127.0.103.67:41715/ using document root <none> and password file <none>
I20250622 13:54:20.981890 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:20.981959 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:20.982079 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:20.982349 7123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34168
I20250622 13:54:20.982738 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root/instance:
uuid: "02e854e07d384b73926562002ae7320f"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:20.985327 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250622 13:54:20.986779 7336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:20.987181 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:20.987280 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root
uuid: "02e854e07d384b73926562002ae7320f"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:20.987381 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:21.006211 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:21.006713 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:21.007200 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:21.007992 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:54:21.008047 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:21.008116 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:54:21.008153 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:21.019219 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:33049
I20250622 13:54:21.019239 7398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:33049 every 8 connection(s)
I20250622 13:54:21.025753 7399 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44101
I20250622 13:54:21.025894 7399 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:21.026469 7399 heartbeater.cc:507] Master 127.0.103.126:44101 requested a full tablet report, sending...
I20250622 13:54:21.027550 7123 ts_manager.cc:194] Registered new tserver with Master: 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:21.028041 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008013051s
I20250622 13:54:21.028348 7123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34178
I20250622 13:54:21.947907 7250 heartbeater.cc:499] Master 127.0.103.126:44101 was elected leader, sending a full tablet report...
I20250622 13:54:21.983826 7325 heartbeater.cc:499] Master 127.0.103.126:44101 was elected leader, sending a full tablet report...
I20250622 13:54:22.029750 7399 heartbeater.cc:499] Master 127.0.103.126:44101 was elected leader, sending a full tablet report...
I20250622 13:54:22.056303 413 test_util.cc:276] Using random seed: 1856977768
I20250622 13:54:22.071270 7123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:54:22.072027 7123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:54:22.095131 7364 tablet_service.cc:1468] Processing CreateTablet for tablet 24006fadf8ac4ea1bae1a772207cd122 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:22.095687 7364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24006fadf8ac4ea1bae1a772207cd122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.095836 7363 tablet_service.cc:1468] Processing CreateTablet for tablet f836cb47622a42ba93df50089655d147 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:22.096241 7363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f836cb47622a42ba93df50089655d147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.096714 7289 tablet_service.cc:1468] Processing CreateTablet for tablet f836cb47622a42ba93df50089655d147 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:22.097072 7289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f836cb47622a42ba93df50089655d147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.097002 7290 tablet_service.cc:1468] Processing CreateTablet for tablet 24006fadf8ac4ea1bae1a772207cd122 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:22.097311 7290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24006fadf8ac4ea1bae1a772207cd122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.097595 7288 tablet_service.cc:1468] Processing CreateTablet for tablet ed5c98b314f340ada36c672fe683ef7a (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:22.097965 7288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5c98b314f340ada36c672fe683ef7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.101673 7287 tablet_service.cc:1468] Processing CreateTablet for tablet 574aacbfc59044629a25bc7c211dc570 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:22.102077 7287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574aacbfc59044629a25bc7c211dc570. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.102742 7364 tablet_service.cc:1468] Processing CreateTablet for tablet ed5c98b314f340ada36c672fe683ef7a (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:22.103085 7364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5c98b314f340ada36c672fe683ef7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.103183 7362 tablet_service.cc:1468] Processing CreateTablet for tablet 574aacbfc59044629a25bc7c211dc570 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:22.103734 7362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574aacbfc59044629a25bc7c211dc570. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.109263 7419 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f: Bootstrap starting.
I20250622 13:54:22.110939 7419 tablet_bootstrap.cc:654] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.121311 7215 tablet_service.cc:1468] Processing CreateTablet for tablet 24006fadf8ac4ea1bae1a772207cd122 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:22.121992 7215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24006fadf8ac4ea1bae1a772207cd122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.121888 7212 tablet_service.cc:1468] Processing CreateTablet for tablet 574aacbfc59044629a25bc7c211dc570 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:22.122365 7212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574aacbfc59044629a25bc7c211dc570. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.125638 7213 tablet_service.cc:1468] Processing CreateTablet for tablet ed5c98b314f340ada36c672fe683ef7a (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:22.126223 7213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5c98b314f340ada36c672fe683ef7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.129881 7214 tablet_service.cc:1468] Processing CreateTablet for tablet f836cb47622a42ba93df50089655d147 (DEFAULT_TABLE table=test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:22.130443 7214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f836cb47622a42ba93df50089655d147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.141561 7422 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:22.143988 7422 tablet_bootstrap.cc:654] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.148437 7422 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: No bootstrap required, opened a new log
I20250622 13:54:22.148630 7422 ts_tablet_manager.cc:1397] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250622 13:54:22.149617 7422 raft_consensus.cc:357] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.149871 7422 raft_consensus.cc:383] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.149945 7422 raft_consensus.cc:738] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.150247 7422 consensus_queue.cc:260] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.153020 7420 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071: Bootstrap starting.
I20250622 13:54:22.153264 7422 ts_tablet_manager.cc:1428] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250622 13:54:22.153944 7422 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:22.154733 7419 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f: No bootstrap required, opened a new log
I20250622 13:54:22.154929 7419 ts_tablet_manager.cc:1397] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f: Time spent bootstrapping tablet: real 0.046s user 0.017s sys 0.027s
I20250622 13:54:22.154975 7420 tablet_bootstrap.cc:654] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.155320 7422 tablet_bootstrap.cc:654] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.155830 7419 raft_consensus.cc:357] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.156129 7419 raft_consensus.cc:383] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.156252 7419 raft_consensus.cc:738] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e854e07d384b73926562002ae7320f, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.156594 7419 consensus_queue.cc:260] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.158758 7419 ts_tablet_manager.cc:1428] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250622 13:54:22.159695 7420 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071: No bootstrap required, opened a new log
I20250622 13:54:22.159837 7422 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: No bootstrap required, opened a new log
I20250622 13:54:22.159832 7420 ts_tablet_manager.cc:1397] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250622 13:54:22.159704 7419 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f: Bootstrap starting.
I20250622 13:54:22.159942 7422 ts_tablet_manager.cc:1397] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250622 13:54:22.160527 7420 raft_consensus.cc:357] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.160697 7420 raft_consensus.cc:383] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.160660 7422 raft_consensus.cc:357] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.160770 7420 raft_consensus.cc:738] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.160852 7422 raft_consensus.cc:383] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.160924 7422 raft_consensus.cc:738] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.161146 7422 consensus_queue.cc:260] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.161126 7420 consensus_queue.cc:260] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.161844 7419 tablet_bootstrap.cc:654] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.162901 7420 ts_tablet_manager.cc:1428] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250622 13:54:22.163440 7420 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071: Bootstrap starting.
I20250622 13:54:22.163684 7422 ts_tablet_manager.cc:1428] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250622 13:54:22.163954 7422 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:22.164063 7419 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f: No bootstrap required, opened a new log
I20250622 13:54:22.164197 7419 ts_tablet_manager.cc:1397] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250622 13:54:22.165009 7420 tablet_bootstrap.cc:654] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.165447 7422 tablet_bootstrap.cc:654] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.164917 7419 raft_consensus.cc:357] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.165611 7419 raft_consensus.cc:383] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.165683 7419 raft_consensus.cc:738] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e854e07d384b73926562002ae7320f, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.165975 7419 consensus_queue.cc:260] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.166486 7419 ts_tablet_manager.cc:1428] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:54:22.166814 7419 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: Bootstrap starting.
I20250622 13:54:22.168606 7419 tablet_bootstrap.cc:654] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.172612 7419 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: No bootstrap required, opened a new log
I20250622 13:54:22.172750 7419 ts_tablet_manager.cc:1397] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250622 13:54:22.172833 7422 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: No bootstrap required, opened a new log
I20250622 13:54:22.172952 7420 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071: No bootstrap required, opened a new log
I20250622 13:54:22.172959 7422 ts_tablet_manager.cc:1397] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20250622 13:54:22.173069 7420 ts_tablet_manager.cc:1397] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20250622 13:54:22.173429 7419 raft_consensus.cc:357] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.173566 7419 raft_consensus.cc:383] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.173622 7419 raft_consensus.cc:738] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e854e07d384b73926562002ae7320f, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.173651 7420 raft_consensus.cc:357] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.173651 7422 raft_consensus.cc:357] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.173837 7420 raft_consensus.cc:383] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.173902 7420 raft_consensus.cc:738] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.173847 7422 raft_consensus.cc:383] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.174031 7422 raft_consensus.cc:738] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.174100 7420 consensus_queue.cc:260] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.173846 7419 consensus_queue.cc:260] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.174180 7422 consensus_queue.cc:260] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.174602 7419 ts_tablet_manager.cc:1428] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:22.174865 7419 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f: Bootstrap starting.
I20250622 13:54:22.176010 7420 ts_tablet_manager.cc:1428] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:54:22.176054 7419 tablet_bootstrap.cc:654] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.176252 7420 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071: Bootstrap starting.
I20250622 13:54:22.177201 7422 ts_tablet_manager.cc:1428] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250622 13:54:22.177476 7422 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:22.178892 7422 tablet_bootstrap.cc:654] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.182683 7420 tablet_bootstrap.cc:654] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.184684 7419 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f: No bootstrap required, opened a new log
I20250622 13:54:22.184684 7420 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071: No bootstrap required, opened a new log
I20250622 13:54:22.184842 7420 ts_tablet_manager.cc:1397] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20250622 13:54:22.184842 7419 ts_tablet_manager.cc:1397] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20250622 13:54:22.185079 7422 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: No bootstrap required, opened a new log
I20250622 13:54:22.185199 7422 ts_tablet_manager.cc:1397] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20250622 13:54:22.185585 7419 raft_consensus.cc:357] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.185585 7420 raft_consensus.cc:357] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.185801 7419 raft_consensus.cc:383] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.185810 7420 raft_consensus.cc:383] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.185789 7422 raft_consensus.cc:357] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.185897 7419 raft_consensus.cc:738] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e854e07d384b73926562002ae7320f, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.185937 7420 raft_consensus.cc:738] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.185954 7422 raft_consensus.cc:383] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.186002 7422 raft_consensus.cc:738] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.186106 7419 consensus_queue.cc:260] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.186148 7420 consensus_queue.cc:260] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.186223 7422 consensus_queue.cc:260] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.186564 7419 ts_tablet_manager.cc:1428] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250622 13:54:22.186573 7420 ts_tablet_manager.cc:1428] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:54:22.186676 7422 ts_tablet_manager.cc:1428] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:22.186841 7420 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: Bootstrap starting.
I20250622 13:54:22.188294 7420 tablet_bootstrap.cc:654] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.191354 7420 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: No bootstrap required, opened a new log
I20250622 13:54:22.191479 7420 ts_tablet_manager.cc:1397] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250622 13:54:22.192008 7420 raft_consensus.cc:357] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.192131 7420 raft_consensus.cc:383] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.192188 7420 raft_consensus.cc:738] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.192363 7420 consensus_queue.cc:260] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.192694 7420 ts_tablet_manager.cc:1428] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:54:22.229614 7427 raft_consensus.cc:491] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:22.229802 7427 raft_consensus.cc:513] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.230589 7427 leader_election.cc:290] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387)
I20250622 13:54:22.236349 7225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f836cb47622a42ba93df50089655d147" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a" is_pre_election: true
I20250622 13:54:22.236351 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f836cb47622a42ba93df50089655d147" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071" is_pre_election: true
I20250622 13:54:22.236578 7225 raft_consensus.cc:2466] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 0.
I20250622 13:54:22.236599 7300 raft_consensus.cc:2466] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 0.
I20250622 13:54:22.237013 7338 leader_election.cc:304] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.237277 7427 raft_consensus.cc:2802] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:22.237378 7427 raft_consensus.cc:491] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:22.237464 7427 raft_consensus.cc:3058] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.239277 7427 raft_consensus.cc:513] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.239744 7427 leader_election.cc:290] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 election: Requested vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387)
I20250622 13:54:22.240164 7225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f836cb47622a42ba93df50089655d147" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a"
I20250622 13:54:22.240236 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f836cb47622a42ba93df50089655d147" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071"
I20250622 13:54:22.240329 7225 raft_consensus.cc:3058] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.240350 7300 raft_consensus.cc:3058] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.242255 7225 raft_consensus.cc:2466] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 1.
I20250622 13:54:22.242264 7300 raft_consensus.cc:2466] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 1.
I20250622 13:54:22.242585 7338 leader_election.cc:304] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071; no voters:
I20250622 13:54:22.242835 7427 raft_consensus.cc:2802] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:22.243191 7427 raft_consensus.cc:695] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Becoming Leader. State: Replica: 02e854e07d384b73926562002ae7320f, State: Running, Role: LEADER
I20250622 13:54:22.243412 7427 consensus_queue.cc:237] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.245642 7122 catalog_manager.cc:5582] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f reported cstate change: term changed from 0 to 1, leader changed from <none> to 02e854e07d384b73926562002ae7320f (127.0.103.67). New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: HEALTHY } } }
I20250622 13:54:22.384192 7428 raft_consensus.cc:491] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:22.384409 7428 raft_consensus.cc:513] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.385188 7428 leader_election.cc:290] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.386742 7424 raft_consensus.cc:491] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:22.386909 7424 raft_consensus.cc:513] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.387805 7424 leader_election.cc:290] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.394512 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed5c98b314f340ada36c672fe683ef7a" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f" is_pre_election: true
I20250622 13:54:22.394795 7374 raft_consensus.cc:2466] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f475d5335b4087a1319af1128b2071 in term 0.
I20250622 13:54:22.394798 7225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed5c98b314f340ada36c672fe683ef7a" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a" is_pre_election: true
I20250622 13:54:22.395278 7225 raft_consensus.cc:2466] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f475d5335b4087a1319af1128b2071 in term 0.
I20250622 13:54:22.395766 7266 leader_election.cc:304] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0f475d5335b4087a1319af1128b2071, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.396104 7428 raft_consensus.cc:2802] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:22.396207 7428 raft_consensus.cc:491] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:22.396310 7428 raft_consensus.cc:3058] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.400290 7428 raft_consensus.cc:513] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.401338 7225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed5c98b314f340ada36c672fe683ef7a" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a"
I20250622 13:54:22.401530 7225 raft_consensus.cc:3058] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.401742 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed5c98b314f340ada36c672fe683ef7a" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f"
I20250622 13:54:22.401925 7374 raft_consensus.cc:3058] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.402253 7373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f" is_pre_election: true
I20250622 13:54:22.402431 7373 raft_consensus.cc:2466] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 0.
I20250622 13:54:22.402619 7428 leader_election.cc:290] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 1 election: Requested vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.402899 7190 leader_election.cc:304] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.403187 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071" is_pre_election: true
I20250622 13:54:22.403290 7424 raft_consensus.cc:2802] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:22.403321 7300 raft_consensus.cc:2466] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 0.
I20250622 13:54:22.403368 7424 raft_consensus.cc:491] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:22.403450 7424 raft_consensus.cc:3058] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.404352 7225 raft_consensus.cc:2466] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f475d5335b4087a1319af1128b2071 in term 1.
I20250622 13:54:22.404443 7374 raft_consensus.cc:2466] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f475d5335b4087a1319af1128b2071 in term 1.
I20250622 13:54:22.404672 7266 leader_election.cc:304] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0f475d5335b4087a1319af1128b2071, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.404937 7428 raft_consensus.cc:2802] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:22.405431 7428 raft_consensus.cc:695] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Becoming Leader. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Running, Role: LEADER
I20250622 13:54:22.405697 7428 consensus_queue.cc:237] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.406085 7424 raft_consensus.cc:513] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.406678 7424 leader_election.cc:290] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 election: Requested vote from peers c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.407428 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f"
I20250622 13:54:22.407717 7374 raft_consensus.cc:3058] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.409058 7122 catalog_manager.cc:5582] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f475d5335b4087a1319af1128b2071 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "c0f475d5335b4087a1319af1128b2071" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:22.410194 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071"
I20250622 13:54:22.410331 7300 raft_consensus.cc:3058] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.411262 7374 raft_consensus.cc:2466] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 1.
I20250622 13:54:22.411688 7190 leader_election.cc:304] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.411968 7424 raft_consensus.cc:2802] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:22.412388 7424 raft_consensus.cc:695] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 LEADER]: Becoming Leader. State: Replica: cc8abf945e6841afb6734436277e357a, State: Running, Role: LEADER
I20250622 13:54:22.412703 7424 consensus_queue.cc:237] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.412990 7300 raft_consensus.cc:2466] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 1.
I20250622 13:54:22.415534 7122 catalog_manager.cc:5582] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8abf945e6841afb6734436277e357a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "cc8abf945e6841afb6734436277e357a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:22.455241 7427 raft_consensus.cc:491] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:22.455471 7427 raft_consensus.cc:513] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.456077 7427 leader_election.cc:290] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387), cc8abf945e6841afb6734436277e357a (127.0.103.65:41121)
I20250622 13:54:22.456629 7225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24006fadf8ac4ea1bae1a772207cd122" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a" is_pre_election: true
I20250622 13:54:22.456666 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24006fadf8ac4ea1bae1a772207cd122" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071" is_pre_election: true
I20250622 13:54:22.456837 7225 raft_consensus.cc:2466] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 0.
I20250622 13:54:22.456862 7300 raft_consensus.cc:2466] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 0.
I20250622 13:54:22.457305 7338 leader_election.cc:304] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071; no voters:
I20250622 13:54:22.457638 7427 raft_consensus.cc:2802] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:22.457751 7427 raft_consensus.cc:491] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:22.457854 7427 raft_consensus.cc:3058] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.459771 7427 raft_consensus.cc:513] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.460261 7427 leader_election.cc:290] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 election: Requested vote from peers c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387), cc8abf945e6841afb6734436277e357a (127.0.103.65:41121)
I20250622 13:54:22.460676 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24006fadf8ac4ea1bae1a772207cd122" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071"
I20250622 13:54:22.460728 7225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24006fadf8ac4ea1bae1a772207cd122" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a"
I20250622 13:54:22.460863 7300 raft_consensus.cc:3058] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.460868 7225 raft_consensus.cc:3058] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.462718 7300 raft_consensus.cc:2466] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 1.
I20250622 13:54:22.462750 7225 raft_consensus.cc:2466] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 1.
I20250622 13:54:22.463080 7338 leader_election.cc:304] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071; no voters:
I20250622 13:54:22.463336 7427 raft_consensus.cc:2802] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:22.463486 7427 raft_consensus.cc:695] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Becoming Leader. State: Replica: 02e854e07d384b73926562002ae7320f, State: Running, Role: LEADER
I20250622 13:54:22.463734 7427 consensus_queue.cc:237] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:22.465919 7122 catalog_manager.cc:5582] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f reported cstate change: term changed from 0 to 1, leader changed from <none> to 02e854e07d384b73926562002ae7320f (127.0.103.67). New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:22.546489 413 test_util.cc:276] Using random seed: 1857467931
I20250622 13:54:22.557790 7122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34186:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 13:54:22.558427 7122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:54:22.565662 7213 tablet_service.cc:1468] Processing CreateTablet for tablet cfff2afb0bc14d3a83aa1bedd156d26d (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:22.565816 7290 tablet_service.cc:1468] Processing CreateTablet for tablet cfff2afb0bc14d3a83aa1bedd156d26d (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:22.566141 7290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfff2afb0bc14d3a83aa1bedd156d26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.566443 7287 tablet_service.cc:1468] Processing CreateTablet for tablet 47688cd9cb0a457383d5a27e813b5c0f (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:22.566839 7287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47688cd9cb0a457383d5a27e813b5c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.571200 7420 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: Bootstrap starting.
I20250622 13:54:22.571835 7362 tablet_service.cc:1468] Processing CreateTablet for tablet 47688cd9cb0a457383d5a27e813b5c0f (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:22.571969 7364 tablet_service.cc:1468] Processing CreateTablet for tablet 531faa077d23417798976a089cc8c882 (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:22.572053 7363 tablet_service.cc:1468] Processing CreateTablet for tablet cfff2afb0bc14d3a83aa1bedd156d26d (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250622 13:54:22.572250 7362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47688cd9cb0a457383d5a27e813b5c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.572436 7363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfff2afb0bc14d3a83aa1bedd156d26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.572901 7420 tablet_bootstrap.cc:654] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.573011 7364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531faa077d23417798976a089cc8c882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.577886 7361 tablet_service.cc:1468] Processing CreateTablet for tablet 6933d3c7479e4845aa3e63e8d0597cd3 (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:22.578460 7361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6933d3c7479e4845aa3e63e8d0597cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.581147 7419 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: Bootstrap starting.
I20250622 13:54:22.582231 7215 tablet_service.cc:1468] Processing CreateTablet for tablet 6933d3c7479e4845aa3e63e8d0597cd3 (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:22.582319 7214 tablet_service.cc:1468] Processing CreateTablet for tablet 47688cd9cb0a457383d5a27e813b5c0f (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250622 13:54:22.582757 7420 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: No bootstrap required, opened a new log
I20250622 13:54:22.582885 7419 tablet_bootstrap.cc:654] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.582914 7212 tablet_service.cc:1468] Processing CreateTablet for tablet 531faa077d23417798976a089cc8c882 (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:22.583186 7420 ts_tablet_manager.cc:1397] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20250622 13:54:22.583521 7289 tablet_service.cc:1468] Processing CreateTablet for tablet 6933d3c7479e4845aa3e63e8d0597cd3 (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250622 13:54:22.583935 7289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6933d3c7479e4845aa3e63e8d0597cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.584352 7420 raft_consensus.cc:357] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.584653 7420 raft_consensus.cc:383] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.584786 7420 raft_consensus.cc:738] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.585155 7420 consensus_queue.cc:260] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.586123 7420 ts_tablet_manager.cc:1428] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250622 13:54:22.566141 7213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfff2afb0bc14d3a83aa1bedd156d26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.586434 7419 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: No bootstrap required, opened a new log
I20250622 13:54:22.586565 7215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6933d3c7479e4845aa3e63e8d0597cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.586759 7420 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: Bootstrap starting.
I20250622 13:54:22.588441 7420 tablet_bootstrap.cc:654] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.592756 7212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531faa077d23417798976a089cc8c882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.593084 7422 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:22.586585 7419 ts_tablet_manager.cc:1397] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250622 13:54:22.594816 7422 tablet_bootstrap.cc:654] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.595248 7419 raft_consensus.cc:357] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.595882 7419 raft_consensus.cc:383] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.595985 7419 raft_consensus.cc:738] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e854e07d384b73926562002ae7320f, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.596302 7419 consensus_queue.cc:260] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.597054 7422 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: No bootstrap required, opened a new log
I20250622 13:54:22.597186 7422 ts_tablet_manager.cc:1397] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250622 13:54:22.597574 7214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47688cd9cb0a457383d5a27e813b5c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.598101 7419 ts_tablet_manager.cc:1428] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250622 13:54:22.598062 7422 raft_consensus.cc:357] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.598228 7422 raft_consensus.cc:383] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.598304 7422 raft_consensus.cc:738] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.598513 7419 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: Bootstrap starting.
I20250622 13:54:22.600824 7419 tablet_bootstrap.cc:654] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.602173 7422 consensus_queue.cc:260] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.602800 7287 tablet_service.cc:1468] Processing CreateTablet for tablet 531faa077d23417798976a089cc8c882 (DEFAULT_TABLE table=dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250622 13:54:22.603363 7287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 531faa077d23417798976a089cc8c882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:22.604884 7422 ts_tablet_manager.cc:1428] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250622 13:54:22.607424 7422 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:22.609206 7422 tablet_bootstrap.cc:654] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.610539 7419 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: No bootstrap required, opened a new log
I20250622 13:54:22.610707 7419 ts_tablet_manager.cc:1397] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20250622 13:54:22.611702 7419 raft_consensus.cc:357] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.611981 7419 raft_consensus.cc:383] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.612118 7419 raft_consensus.cc:738] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e854e07d384b73926562002ae7320f, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.612408 7419 consensus_queue.cc:260] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.612939 7420 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: No bootstrap required, opened a new log
I20250622 13:54:22.613058 7419 ts_tablet_manager.cc:1428] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:22.613260 7422 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: No bootstrap required, opened a new log
I20250622 13:54:22.613395 7419 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: Bootstrap starting.
I20250622 13:54:22.613425 7422 ts_tablet_manager.cc:1397] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20250622 13:54:22.613085 7420 ts_tablet_manager.cc:1397] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.009s
I20250622 13:54:22.614212 7422 raft_consensus.cc:357] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.614529 7422 raft_consensus.cc:383] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.614639 7422 raft_consensus.cc:738] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.614893 7422 consensus_queue.cc:260] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.615218 7419 tablet_bootstrap.cc:654] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.615406 7422 ts_tablet_manager.cc:1428] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250622 13:54:22.615429 7420 raft_consensus.cc:357] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.615770 7420 raft_consensus.cc:383] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.615798 7422 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:22.615939 7420 raft_consensus.cc:738] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.616210 7420 consensus_queue.cc:260] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.616837 7420 ts_tablet_manager.cc:1428] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:54:22.617167 7420 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: Bootstrap starting.
I20250622 13:54:22.618317 7422 tablet_bootstrap.cc:654] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.619108 7420 tablet_bootstrap.cc:654] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.621078 7427 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:22.630187 7422 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: No bootstrap required, opened a new log
I20250622 13:54:22.630471 7422 ts_tablet_manager.cc:1397] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20250622 13:54:22.631558 7422 raft_consensus.cc:357] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.632287 7422 raft_consensus.cc:383] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.632419 7422 raft_consensus.cc:738] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.632812 7422 consensus_queue.cc:260] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.633543 7422 ts_tablet_manager.cc:1428] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250622 13:54:22.633982 7422 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:22.636315 7422 tablet_bootstrap.cc:654] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.636693 7439 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:22.637782 7420 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: No bootstrap required, opened a new log
I20250622 13:54:22.637956 7420 ts_tablet_manager.cc:1397] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20250622 13:54:22.638623 7419 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: No bootstrap required, opened a new log
I20250622 13:54:22.638839 7419 ts_tablet_manager.cc:1397] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.006s
I20250622 13:54:22.638809 7420 raft_consensus.cc:357] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.639001 7420 raft_consensus.cc:383] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.639093 7420 raft_consensus.cc:738] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.639341 7420 consensus_queue.cc:260] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.639837 7420 ts_tablet_manager.cc:1428] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:54:22.640000 7419 raft_consensus.cc:357] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.640233 7419 raft_consensus.cc:383] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.640362 7419 raft_consensus.cc:738] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e854e07d384b73926562002ae7320f, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.640678 7419 consensus_queue.cc:260] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.641309 7419 ts_tablet_manager.cc:1428] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250622 13:54:22.641754 7419 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: Bootstrap starting.
I20250622 13:54:22.644174 7419 tablet_bootstrap.cc:654] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.640138 7420 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: Bootstrap starting.
I20250622 13:54:22.647679 7420 tablet_bootstrap.cc:654] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: Neither blocks nor log segments found. Creating new log.
I20250622 13:54:22.653322 7419 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: No bootstrap required, opened a new log
I20250622 13:54:22.653587 7419 ts_tablet_manager.cc:1397] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20250622 13:54:22.654685 7419 raft_consensus.cc:357] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.654973 7419 raft_consensus.cc:383] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.655081 7419 raft_consensus.cc:738] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e854e07d384b73926562002ae7320f, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.655407 7419 consensus_queue.cc:260] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.656013 7419 ts_tablet_manager.cc:1428] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250622 13:54:22.658993 7422 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: No bootstrap required, opened a new log
I20250622 13:54:22.659160 7422 ts_tablet_manager.cc:1397] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.011s
I20250622 13:54:22.659473 7420 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: No bootstrap required, opened a new log
I20250622 13:54:22.659672 7420 ts_tablet_manager.cc:1397] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20250622 13:54:22.660025 7422 raft_consensus.cc:357] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.660233 7422 raft_consensus.cc:383] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.660322 7422 raft_consensus.cc:738] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.660626 7422 consensus_queue.cc:260] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.661065 7420 raft_consensus.cc:357] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.661265 7420 raft_consensus.cc:383] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:54:22.661378 7420 raft_consensus.cc:738] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Initialized, Role: FOLLOWER
I20250622 13:54:22.661621 7420 consensus_queue.cc:260] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.662256 7420 ts_tablet_manager.cc:1428] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250622 13:54:22.662915 7422 ts_tablet_manager.cc:1428] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250622 13:54:22.755995 7424 raft_consensus.cc:491] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:22.756254 7424 raft_consensus.cc:513] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.756999 7424 leader_election.cc:290] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.757550 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47688cd9cb0a457383d5a27e813b5c0f" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071" is_pre_election: true
I20250622 13:54:22.757565 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47688cd9cb0a457383d5a27e813b5c0f" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f" is_pre_election: true
I20250622 13:54:22.757787 7300 raft_consensus.cc:2466] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 0.
I20250622 13:54:22.757860 7374 raft_consensus.cc:2466] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 0.
I20250622 13:54:22.758224 7189 leader_election.cc:304] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0f475d5335b4087a1319af1128b2071, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.758536 7424 raft_consensus.cc:2802] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:22.758620 7424 raft_consensus.cc:491] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:22.758682 7424 raft_consensus.cc:3058] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.760535 7424 raft_consensus.cc:513] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.761047 7424 leader_election.cc:290] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 election: Requested vote from peers c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.761554 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47688cd9cb0a457383d5a27e813b5c0f" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f"
I20250622 13:54:22.761565 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47688cd9cb0a457383d5a27e813b5c0f" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071"
I20250622 13:54:22.761726 7374 raft_consensus.cc:3058] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.761759 7300 raft_consensus.cc:3058] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.763718 7374 raft_consensus.cc:2466] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 1.
I20250622 13:54:22.763809 7300 raft_consensus.cc:2466] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 1.
I20250622 13:54:22.764256 7189 leader_election.cc:304] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.764598 7424 raft_consensus.cc:2802] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:22.764775 7424 raft_consensus.cc:695] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 LEADER]: Becoming Leader. State: Replica: cc8abf945e6841afb6734436277e357a, State: Running, Role: LEADER
I20250622 13:54:22.765064 7424 consensus_queue.cc:237] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.767777 7121 catalog_manager.cc:5582] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8abf945e6841afb6734436277e357a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "cc8abf945e6841afb6734436277e357a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:22.778771 7439 raft_consensus.cc:491] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:22.778957 7428 raft_consensus.cc:491] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:22.779004 7439 raft_consensus.cc:513] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.779136 7428 raft_consensus.cc:513] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.779728 7428 leader_election.cc:290] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.779729 7439 leader_election.cc:290] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387)
I20250622 13:54:22.780203 7225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6933d3c7479e4845aa3e63e8d0597cd3" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a" is_pre_election: true
I20250622 13:54:22.780220 7224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "531faa077d23417798976a089cc8c882" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a" is_pre_election: true
I20250622 13:54:22.780203 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6933d3c7479e4845aa3e63e8d0597cd3" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071" is_pre_election: true
I20250622 13:54:22.780550 7224 raft_consensus.cc:2466] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f475d5335b4087a1319af1128b2071 in term 0.
I20250622 13:54:22.780547 7300 raft_consensus.cc:2466] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 0.
I20250622 13:54:22.780550 7225 raft_consensus.cc:2466] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 0.
I20250622 13:54:22.780220 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "531faa077d23417798976a089cc8c882" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f" is_pre_election: true
I20250622 13:54:22.780970 7374 raft_consensus.cc:2466] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f475d5335b4087a1319af1128b2071 in term 0.
I20250622 13:54:22.781059 7340 leader_election.cc:304] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.781056 7266 leader_election.cc:304] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0f475d5335b4087a1319af1128b2071, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.781397 7428 raft_consensus.cc:2802] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:22.781497 7439 raft_consensus.cc:2802] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:22.781510 7428 raft_consensus.cc:491] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:22.781637 7439 raft_consensus.cc:491] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:22.781664 7428 raft_consensus.cc:3058] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.781757 7439 raft_consensus.cc:3058] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.783568 7428 raft_consensus.cc:513] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.783571 7439 raft_consensus.cc:513] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.784111 7439 leader_election.cc:290] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 election: Requested vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387)
I20250622 13:54:22.784236 7428 leader_election.cc:290] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 1 election: Requested vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.784570 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "531faa077d23417798976a089cc8c882" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f"
I20250622 13:54:22.784572 7225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6933d3c7479e4845aa3e63e8d0597cd3" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a"
I20250622 13:54:22.784768 7374 raft_consensus.cc:3058] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.784798 7225 raft_consensus.cc:3058] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.784819 7224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "531faa077d23417798976a089cc8c882" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a"
I20250622 13:54:22.785033 7224 raft_consensus.cc:3058] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.786857 7374 raft_consensus.cc:2466] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f475d5335b4087a1319af1128b2071 in term 1.
I20250622 13:54:22.787240 7265 leader_election.cc:304] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071; no voters:
I20250622 13:54:22.787475 7428 raft_consensus.cc:2802] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:22.787549 7225 raft_consensus.cc:2466] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 1.
I20250622 13:54:22.787609 7224 raft_consensus.cc:2466] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f475d5335b4087a1319af1128b2071 in term 1.
I20250622 13:54:22.787633 7428 raft_consensus.cc:695] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Becoming Leader. State: Replica: c0f475d5335b4087a1319af1128b2071, State: Running, Role: LEADER
I20250622 13:54:22.787894 7428 consensus_queue.cc:237] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.787981 7340 leader_election.cc:304] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.784854 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6933d3c7479e4845aa3e63e8d0597cd3" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071"
I20250622 13:54:22.788281 7300 raft_consensus.cc:3058] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.788338 7439 raft_consensus.cc:2802] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:22.788467 7439 raft_consensus.cc:695] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Becoming Leader. State: Replica: 02e854e07d384b73926562002ae7320f, State: Running, Role: LEADER
I20250622 13:54:22.788743 7439 consensus_queue.cc:237] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.791215 7121 catalog_manager.cc:5582] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f475d5335b4087a1319af1128b2071 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "c0f475d5335b4087a1319af1128b2071" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:22.791972 7300 raft_consensus.cc:2466] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 1.
I20250622 13:54:22.793975 7122 catalog_manager.cc:5582] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f reported cstate change: term changed from 0 to 1, leader changed from <none> to 02e854e07d384b73926562002ae7320f (127.0.103.67). New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: HEALTHY } } }
I20250622 13:54:22.821687 7424 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:22.825491 7424 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:22.836426 7428 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:22.841641 7480 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:54:22.867966 7171 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250622 13:54:22.895587 7448 raft_consensus.cc:491] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:54:22.895836 7448 raft_consensus.cc:513] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.896592 7448 leader_election.cc:290] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.897032 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfff2afb0bc14d3a83aa1bedd156d26d" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071" is_pre_election: true
I20250622 13:54:22.897078 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfff2afb0bc14d3a83aa1bedd156d26d" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f" is_pre_election: true
I20250622 13:54:22.897241 7374 raft_consensus.cc:2466] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 0.
I20250622 13:54:22.897244 7300 raft_consensus.cc:2466] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 0.
I20250622 13:54:22.897681 7190 leader_election.cc:304] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.897953 7448 raft_consensus.cc:2802] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:54:22.898054 7448 raft_consensus.cc:491] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:54:22.898149 7448 raft_consensus.cc:3058] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.900197 7448 raft_consensus.cc:513] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.900715 7448 leader_election.cc:290] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 election: Requested vote from peers c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:22.901252 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfff2afb0bc14d3a83aa1bedd156d26d" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071"
I20250622 13:54:22.901258 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfff2afb0bc14d3a83aa1bedd156d26d" candidate_uuid: "cc8abf945e6841afb6734436277e357a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f"
I20250622 13:54:22.901505 7374 raft_consensus.cc:3058] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.901513 7300 raft_consensus.cc:3058] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:54:22.904286 7374 raft_consensus.cc:2466] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 1.
I20250622 13:54:22.904281 7300 raft_consensus.cc:2466] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8abf945e6841afb6734436277e357a in term 1.
I20250622 13:54:22.904800 7189 leader_election.cc:304] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071, cc8abf945e6841afb6734436277e357a; no voters:
I20250622 13:54:22.905103 7448 raft_consensus.cc:2802] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:54:22.905256 7448 raft_consensus.cc:695] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 LEADER]: Becoming Leader. State: Replica: cc8abf945e6841afb6734436277e357a, State: Running, Role: LEADER
I20250622 13:54:22.905571 7448 consensus_queue.cc:237] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:22.908390 7122 catalog_manager.cc:5582] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8abf945e6841afb6734436277e357a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "cc8abf945e6841afb6734436277e357a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:22.912114 7439 consensus_queue.cc:1035] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:22.916960 7439 consensus_queue.cc:1035] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:54:22.981215 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250622 13:54:22.988921 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:22.989341 413 tablet_replica.cc:331] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: stopping tablet replica
I20250622 13:54:22.989640 413 raft_consensus.cc:2241] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:22.989923 413 pending_rounds.cc:70] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Trying to abort 1 pending ops.
I20250622 13:54:22.990003 413 pending_rounds.cc:77] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459495488118784 op_type: NO_OP noop_request { }
I20250622 13:54:22.990126 413 raft_consensus.cc:2887] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:54:22.990252 413 raft_consensus.cc:2270] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:22.991134 413 tablet_replica.cc:331] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: stopping tablet replica
I20250622 13:54:22.991315 413 raft_consensus.cc:2241] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:22.991461 413 raft_consensus.cc:2270] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:22.991991 413 tablet_replica.cc:331] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: stopping tablet replica
I20250622 13:54:22.992132 413 raft_consensus.cc:2241] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:22.992429 413 raft_consensus.cc:2270] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:22.992967 413 tablet_replica.cc:331] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: stopping tablet replica
I20250622 13:54:22.993153 413 raft_consensus.cc:2241] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:22.993279 413 raft_consensus.cc:2270] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:22.993923 413 tablet_replica.cc:331] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: stopping tablet replica
I20250622 13:54:22.994057 413 raft_consensus.cc:2241] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:22.994206 413 raft_consensus.cc:2270] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:22.994733 413 tablet_replica.cc:331] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: stopping tablet replica
I20250622 13:54:22.994889 413 raft_consensus.cc:2241] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:22.995030 413 raft_consensus.cc:2270] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:22.995688 413 tablet_replica.cc:331] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: stopping tablet replica
I20250622 13:54:22.995832 413 raft_consensus.cc:2241] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:22.996037 413 pending_rounds.cc:70] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Trying to abort 1 pending ops.
I20250622 13:54:22.996111 413 pending_rounds.cc:77] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459496063889408 op_type: NO_OP noop_request { }
I20250622 13:54:22.996232 413 raft_consensus.cc:2887] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:54:22.996307 413 raft_consensus.cc:2270] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:22.996922 413 tablet_replica.cc:331] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: stopping tablet replica
I20250622 13:54:22.997025 413 raft_consensus.cc:2241] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:22.997144 413 raft_consensus.cc:2270] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:23.012493 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
W20250622 13:54:23.161399 7340 consensus_peers.cc:487] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f -> Peer cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): Couldn't send request to peer cc8abf945e6841afb6734436277e357a. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:23.268663 7427 consensus_queue.cc:1035] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:54:23.269709 7266 consensus_peers.cc:487] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 -> Peer cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): Couldn't send request to peer cc8abf945e6841afb6734436277e357a. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250622 13:54:23.272969 7340 consensus_peers.cc:487] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f -> Peer cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): Couldn't send request to peer cc8abf945e6841afb6734436277e357a. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:23.299882 7480 consensus_queue.cc:1035] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:54:23.306087 7266 consensus_peers.cc:487] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 -> Peer cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): Couldn't send request to peer cc8abf945e6841afb6734436277e357a. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250622 13:54:23.374574 7340 consensus_peers.cc:487] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f -> Peer cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): Couldn't send request to peer cc8abf945e6841afb6734436277e357a. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:23.746773 7427 consensus_queue.cc:579] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Leader has been unable to successfully communicate with peer cc8abf945e6841afb6734436277e357a for more than 1 seconds (1.109s)
W20250622 13:54:23.747809 7340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41121: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20250622 13:54:23.868276 7480 consensus_queue.cc:579] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [LEADER]: Leader has been unable to successfully communicate with peer cc8abf945e6841afb6734436277e357a for more than 1 seconds (1.080s)
I20250622 13:54:23.877713 7427 consensus_queue.cc:579] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [LEADER]: Leader has been unable to successfully communicate with peer cc8abf945e6841afb6734436277e357a for more than 1 seconds (1.089s)
I20250622 13:54:24.032665 7121 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250622 13:54:24.032862 7121 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250622 13:54:24.037784 7121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: Sending DeleteTablet for 3 replicas of tablet cfff2afb0bc14d3a83aa1bedd156d26d
I20250622 13:54:24.038242 7121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: Sending DeleteTablet for 3 replicas of tablet 47688cd9cb0a457383d5a27e813b5c0f
I20250622 13:54:24.038601 7287 tablet_service.cc:1515] Processing DeleteTablet for tablet cfff2afb0bc14d3a83aa1bedd156d26d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:40126
I20250622 13:54:24.038632 7363 tablet_service.cc:1515] Processing DeleteTablet for tablet cfff2afb0bc14d3a83aa1bedd156d26d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:38692
I20250622 13:54:24.039050 7361 tablet_service.cc:1515] Processing DeleteTablet for tablet 47688cd9cb0a457383d5a27e813b5c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:38692
I20250622 13:54:24.038659 7121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: Sending DeleteTablet for 3 replicas of tablet 531faa077d23417798976a089cc8c882
I20250622 13:54:24.039372 7289 tablet_service.cc:1515] Processing DeleteTablet for tablet 47688cd9cb0a457383d5a27e813b5c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:40126
I20250622 13:54:24.039737 7121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: Sending DeleteTablet for 3 replicas of tablet 6933d3c7479e4845aa3e63e8d0597cd3
I20250622 13:54:24.040055 7502 tablet_replica.cc:331] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: stopping tablet replica
I20250622 13:54:24.040287 7363 tablet_service.cc:1515] Processing DeleteTablet for tablet 531faa077d23417798976a089cc8c882 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:38692
I20250622 13:54:24.040647 7289 tablet_service.cc:1515] Processing DeleteTablet for tablet 531faa077d23417798976a089cc8c882 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:40126
I20250622 13:54:24.040838 7501 tablet_replica.cc:331] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: stopping tablet replica
I20250622 13:54:24.040854 7290 tablet_service.cc:1515] Processing DeleteTablet for tablet 6933d3c7479e4845aa3e63e8d0597cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:40126
I20250622 13:54:24.041056 7501 raft_consensus.cc:2241] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.041244 7501 raft_consensus.cc:2270] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.041456 7363 tablet_service.cc:1515] Processing DeleteTablet for tablet 6933d3c7479e4845aa3e63e8d0597cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:38692
I20250622 13:54:24.040280 7502 raft_consensus.cc:2241] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.041610 7502 raft_consensus.cc:2270] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250622 13:54:24.041829 7110 catalog_manager.cc:4671] TS cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47688cd9cb0a457383d5a27e813b5c0f: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41121: connect: Connection refused (error 111)
I20250622 13:54:24.042033 7501 ts_tablet_manager.cc:1905] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.042351 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250622 13:54:24.042414 7502 ts_tablet_manager.cc:1905] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250622 13:54:24.044934 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:24.046882 7501 ts_tablet_manager.cc:1918] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250622 13:54:24.047016 7501 log.cc:1199] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root/wals/cfff2afb0bc14d3a83aa1bedd156d26d
I20250622 13:54:24.047361 7501 ts_tablet_manager.cc:1939] T cfff2afb0bc14d3a83aa1bedd156d26d P c0f475d5335b4087a1319af1128b2071: Deleting consensus metadata
I20250622 13:54:24.048586 7501 tablet_replica.cc:331] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: stopping tablet replica
I20250622 13:54:24.048718 7108 catalog_manager.cc:4928] TS c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387): tablet cfff2afb0bc14d3a83aa1bedd156d26d (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.048811 7501 raft_consensus.cc:2241] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.048954 7501 raft_consensus.cc:2270] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.049535 7501 ts_tablet_manager.cc:1905] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.050143 7502 ts_tablet_manager.cc:1918] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250622 13:54:24.050276 7502 log.cc:1199] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root/wals/cfff2afb0bc14d3a83aa1bedd156d26d
I20250622 13:54:24.050675 7502 ts_tablet_manager.cc:1939] T cfff2afb0bc14d3a83aa1bedd156d26d P 02e854e07d384b73926562002ae7320f: Deleting consensus metadata
I20250622 13:54:24.051674 7502 tablet_replica.cc:331] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: stopping tablet replica
I20250622 13:54:24.051824 7502 raft_consensus.cc:2241] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.051831 7109 catalog_manager.cc:4928] TS 02e854e07d384b73926562002ae7320f (127.0.103.67:33049): tablet cfff2afb0bc14d3a83aa1bedd156d26d (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.051968 7502 raft_consensus.cc:2270] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.052497 7502 ts_tablet_manager.cc:1905] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.055457 7501 ts_tablet_manager.cc:1918] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250622 13:54:24.055636 7501 log.cc:1199] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root/wals/47688cd9cb0a457383d5a27e813b5c0f
I20250622 13:54:24.056005 7501 ts_tablet_manager.cc:1939] T 47688cd9cb0a457383d5a27e813b5c0f P c0f475d5335b4087a1319af1128b2071: Deleting consensus metadata
I20250622 13:54:24.057010 7108 catalog_manager.cc:4928] TS c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387): tablet 47688cd9cb0a457383d5a27e813b5c0f (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.057344 7501 tablet_replica.cc:331] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: stopping tablet replica
I20250622 13:54:24.057549 7501 raft_consensus.cc:2241] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:24.058027 7501 raft_consensus.cc:2270] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250622 13:54:24.050659 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:24.058416 7502 ts_tablet_manager.cc:1918] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250622 13:54:24.058573 7502 log.cc:1199] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root/wals/47688cd9cb0a457383d5a27e813b5c0f
I20250622 13:54:24.058969 7502 ts_tablet_manager.cc:1939] T 47688cd9cb0a457383d5a27e813b5c0f P 02e854e07d384b73926562002ae7320f: Deleting consensus metadata
W20250622 13:54:24.059024 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:24.059051 7501 ts_tablet_manager.cc:1905] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.059762 7502 tablet_replica.cc:331] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: stopping tablet replica
I20250622 13:54:24.059911 7109 catalog_manager.cc:4928] TS 02e854e07d384b73926562002ae7320f (127.0.103.67:33049): tablet 47688cd9cb0a457383d5a27e813b5c0f (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.059994 7502 raft_consensus.cc:2241] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.060319 413 server_base.cc:1048] running on GCE node
I20250622 13:54:24.060433 7502 raft_consensus.cc:2270] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.060823 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:24.060911 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:24.060977 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600464060969 us; error 0 us; skew 500 ppm
I20250622 13:54:24.061175 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:24.062001 7502 ts_tablet_manager.cc:1905] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.062727 413 webserver.cc:469] Webserver started at http://127.0.103.68:36867/ using document root <none> and password file <none>
I20250622 13:54:24.062948 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:24.063011 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:54:24.063139 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:54:24.063716 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-3-root/instance:
uuid: "3f0bb5ab0b90452ba99796f322156573"
format_stamp: "Formatted at 2025-06-22 13:54:24 on dist-test-slave-tmv0"
I20250622 13:54:24.066665 7501 ts_tablet_manager.cc:1918] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250622 13:54:24.066805 7501 log.cc:1199] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root/wals/531faa077d23417798976a089cc8c882
I20250622 13:54:24.067189 7501 ts_tablet_manager.cc:1939] T 531faa077d23417798976a089cc8c882 P c0f475d5335b4087a1319af1128b2071: Deleting consensus metadata
I20250622 13:54:24.068096 7501 tablet_replica.cc:331] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: stopping tablet replica
I20250622 13:54:24.068265 7108 catalog_manager.cc:4928] TS c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387): tablet 531faa077d23417798976a089cc8c882 (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.068300 7501 raft_consensus.cc:2241] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.068449 7501 raft_consensus.cc:2270] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.068096 7502 ts_tablet_manager.cc:1918] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250622 13:54:24.068611 413 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20250622 13:54:24.068699 7502 log.cc:1199] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root/wals/531faa077d23417798976a089cc8c882
I20250622 13:54:24.069105 7502 ts_tablet_manager.cc:1939] T 531faa077d23417798976a089cc8c882 P 02e854e07d384b73926562002ae7320f: Deleting consensus metadata
I20250622 13:54:24.069245 7501 ts_tablet_manager.cc:1905] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.070022 7502 tablet_replica.cc:331] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: stopping tablet replica
I20250622 13:54:24.070086 7109 catalog_manager.cc:4928] TS 02e854e07d384b73926562002ae7320f (127.0.103.67:33049): tablet 531faa077d23417798976a089cc8c882 (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.070216 7502 raft_consensus.cc:2241] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:24.070542 7502 raft_consensus.cc:2270] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.071252 7502 ts_tablet_manager.cc:1905] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.073648 7512 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:24.074036 413 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250622 13:54:24.074131 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-3-root
uuid: "3f0bb5ab0b90452ba99796f322156573"
format_stamp: "Formatted at 2025-06-22 13:54:24 on dist-test-slave-tmv0"
I20250622 13:54:24.074249 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:24.075223 7501 ts_tablet_manager.cc:1918] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250622 13:54:24.075380 7501 log.cc:1199] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root/wals/6933d3c7479e4845aa3e63e8d0597cd3
I20250622 13:54:24.075806 7501 ts_tablet_manager.cc:1939] T 6933d3c7479e4845aa3e63e8d0597cd3 P c0f475d5335b4087a1319af1128b2071: Deleting consensus metadata
I20250622 13:54:24.076783 7108 catalog_manager.cc:4928] TS c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387): tablet 6933d3c7479e4845aa3e63e8d0597cd3 (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.077153 7502 ts_tablet_manager.cc:1918] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250622 13:54:24.077241 7502 log.cc:1199] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root/wals/6933d3c7479e4845aa3e63e8d0597cd3
I20250622 13:54:24.077577 7502 ts_tablet_manager.cc:1939] T 6933d3c7479e4845aa3e63e8d0597cd3 P 02e854e07d384b73926562002ae7320f: Deleting consensus metadata
I20250622 13:54:24.078357 7109 catalog_manager.cc:4928] TS 02e854e07d384b73926562002ae7320f (127.0.103.67:33049): tablet 6933d3c7479e4845aa3e63e8d0597cd3 (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.082399 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:24.082885 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:24.083391 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:24.084198 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:54:24.084254 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:24.084342 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:54:24.084388 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:24.097954 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:33189
I20250622 13:54:24.097992 7574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:33189 every 8 connection(s)
I20250622 13:54:24.099311 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:54:24.103802 7579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:24.103983 7578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:54:24.107254 7582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:54:24.107350 413 server_base.cc:1048] running on GCE node
I20250622 13:54:24.107586 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:54:24.107671 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:54:24.107713 413 hybrid_clock.cc:648] HybridClock initialized: now 1750600464107710 us; error 0 us; skew 500 ppm
I20250622 13:54:24.107880 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:54:24.109275 413 webserver.cc:469] Webserver started at http://127.0.103.65:34253/ using document root <none> and password file <none>
I20250622 13:54:24.109566 413 fs_manager.cc:362] Metadata directory not provided
I20250622 13:54:24.109663 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250622 13:54:24.112208 413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20250622 13:54:24.112681 7575 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44101
I20250622 13:54:24.112888 7575 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:24.113287 7575 heartbeater.cc:507] Master 127.0.103.126:44101 requested a full tablet report, sending...
I20250622 13:54:24.114112 7121 ts_manager.cc:194] Registered new tserver with Master: 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189)
I20250622 13:54:24.114346 7586 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:54:24.114706 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:24.114820 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root
uuid: "cc8abf945e6841afb6734436277e357a"
format_stamp: "Formatted at 2025-06-22 13:54:20 on dist-test-slave-tmv0"
I20250622 13:54:24.114941 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:54:24.115024 7121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34198
I20250622 13:54:24.122889 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:54:24.123399 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:54:24.123939 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:54:24.125264 7593 ts_tablet_manager.cc:542] Loading tablet metadata (0/8 complete)
I20250622 13:54:24.140290 413 ts_tablet_manager.cc:579] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20250622 13:54:24.140388 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20250622 13:54:24.140455 413 ts_tablet_manager.cc:594] Registering tablets (0/8 complete)
I20250622 13:54:24.142376 7593 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:24.146347 7593 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:24.146673 7593 tablet_bootstrap.cc:492] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Bootstrap complete.
I20250622 13:54:24.146925 7593 ts_tablet_manager.cc:1397] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250622 13:54:24.147567 7593 raft_consensus.cc:357] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.147738 7593 raft_consensus.cc:738] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:24.147943 7593 consensus_queue.cc:260] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.148618 7593 ts_tablet_manager.cc:1428] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:24.148845 7593 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:24.152511 7593 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:24.152825 7593 tablet_bootstrap.cc:492] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Bootstrap complete.
I20250622 13:54:24.153059 7593 ts_tablet_manager.cc:1397] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250622 13:54:24.153702 7593 raft_consensus.cc:357] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.153870 7593 raft_consensus.cc:738] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:24.154086 7593 consensus_queue.cc:260] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.154528 7593 ts_tablet_manager.cc:1428] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:24.154747 7593 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:24.158699 413 ts_tablet_manager.cc:610] Registered 8 tablets
I20250622 13:54:24.158794 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.018s user 0.017s sys 0.000s
I20250622 13:54:24.159337 7593 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:24.159780 7593 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Bootstrap complete.
I20250622 13:54:24.160027 7593 ts_tablet_manager.cc:1397] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250622 13:54:24.160782 7593 raft_consensus.cc:357] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.160997 7593 raft_consensus.cc:738] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:24.161274 7593 consensus_queue.cc:260] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.162076 7593 ts_tablet_manager.cc:1428] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250622 13:54:24.162359 7593 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:24.168287 7593 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:24.168661 7593 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Bootstrap complete.
I20250622 13:54:24.168893 7593 ts_tablet_manager.cc:1397] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250622 13:54:24.169530 7593 raft_consensus.cc:357] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.169727 7593 raft_consensus.cc:738] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:24.169976 7593 consensus_queue.cc:260] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.170414 7593 ts_tablet_manager.cc:1428] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250622 13:54:24.170625 7593 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:24.174587 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41121
I20250622 13:54:24.174587 7655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41121 every 8 connection(s)
I20250622 13:54:24.175283 7593 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:24.175674 7593 tablet_bootstrap.cc:492] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Bootstrap complete.
I20250622 13:54:24.175976 7593 ts_tablet_manager.cc:1397] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250622 13:54:24.176647 7593 raft_consensus.cc:357] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:24.176829 7593 raft_consensus.cc:738] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:24.177029 7593 consensus_queue.cc:260] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:24.177521 7593 ts_tablet_manager.cc:1428] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250622 13:54:24.177765 7593 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:24.186700 7656 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44101
I20250622 13:54:24.186863 7656 heartbeater.cc:461] Registering TS with master...
I20250622 13:54:24.187178 7593 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250622 13:54:24.187218 7656 heartbeater.cc:507] Master 127.0.103.126:44101 requested a full tablet report, sending...
I20250622 13:54:24.187626 7593 tablet_bootstrap.cc:492] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Bootstrap complete.
I20250622 13:54:24.187883 7593 ts_tablet_manager.cc:1397] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20250622 13:54:24.188736 7121 ts_manager.cc:194] Re-registered known tserver with Master: cc8abf945e6841afb6734436277e357a (127.0.103.65:41121)
I20250622 13:54:24.188679 7593 raft_consensus.cc:357] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.188943 7593 raft_consensus.cc:738] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:24.189180 7593 consensus_queue.cc:260] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.189616 7593 ts_tablet_manager.cc:1428] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250622 13:54:24.189853 7593 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:24.192670 7121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34212
I20250622 13:54:24.193401 7619 tablet_service.cc:1515] Processing DeleteTablet for tablet 47688cd9cb0a457383d5a27e813b5c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:57030
I20250622 13:54:24.193429 7618 tablet_service.cc:1515] Processing DeleteTablet for tablet 6933d3c7479e4845aa3e63e8d0597cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:57030
I20250622 13:54:24.194267 7656 heartbeater.cc:499] Master 127.0.103.126:44101 was elected leader, sending a full tablet report...
I20250622 13:54:24.194572 7662 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:24.194757 7662 raft_consensus.cc:2241] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.194922 7662 raft_consensus.cc:2270] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250622 13:54:24.195340 7110 catalog_manager.cc:4908] TS cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): delete failed for tablet 47688cd9cb0a457383d5a27e813b5c0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47688cd9cb0a457383d5a27e813b5c0f already in progress: opening tablet
I20250622 13:54:24.196120 7662 ts_tablet_manager.cc:1905] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.197027 7593 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250622 13:54:24.197525 7593 tablet_bootstrap.cc:492] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Bootstrap complete.
I20250622 13:54:24.197775 7593 ts_tablet_manager.cc:1397] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250622 13:54:24.198606 7593 raft_consensus.cc:357] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.198868 7593 raft_consensus.cc:738] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:24.199087 7593 consensus_queue.cc:260] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.199457 7619 tablet_service.cc:1515] Processing DeleteTablet for tablet 47688cd9cb0a457383d5a27e813b5c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:57030
I20250622 13:54:24.199638 7593 ts_tablet_manager.cc:1428] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250622 13:54:24.199826 7593 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Bootstrap starting.
I20250622 13:54:24.201560 7662 ts_tablet_manager.cc:1918] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250622 13:54:24.201658 7662 log.cc:1199] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root/wals/6933d3c7479e4845aa3e63e8d0597cd3
I20250622 13:54:24.202005 7662 ts_tablet_manager.cc:1939] T 6933d3c7479e4845aa3e63e8d0597cd3 P cc8abf945e6841afb6734436277e357a: Deleting consensus metadata
I20250622 13:54:24.202711 7662 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:24.202822 7110 catalog_manager.cc:4928] TS cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): tablet 6933d3c7479e4845aa3e63e8d0597cd3 (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.202877 7662 raft_consensus.cc:2241] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.203006 7662 pending_rounds.cc:70] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Trying to abort 1 pending ops.
I20250622 13:54:24.203073 7662 pending_rounds.cc:77] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459495488118784 op_type: NO_OP noop_request { }
I20250622 13:54:24.203187 7662 raft_consensus.cc:2887] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:54:24.203284 7662 raft_consensus.cc:2270] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.203954 7662 ts_tablet_manager.cc:1905] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.204457 7593 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:24.204792 7593 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Bootstrap complete.
I20250622 13:54:24.205017 7593 ts_tablet_manager.cc:1397] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250622 13:54:24.205574 7593 raft_consensus.cc:357] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.205693 7593 raft_consensus.cc:738] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8abf945e6841afb6734436277e357a, State: Initialized, Role: FOLLOWER
I20250622 13:54:24.205891 7593 consensus_queue.cc:260] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.206284 7593 ts_tablet_manager.cc:1428] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250622 13:54:24.208971 7662 ts_tablet_manager.cc:1918] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250622 13:54:24.209092 7662 log.cc:1199] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root/wals/47688cd9cb0a457383d5a27e813b5c0f
I20250622 13:54:24.209450 7662 ts_tablet_manager.cc:1939] T 47688cd9cb0a457383d5a27e813b5c0f P cc8abf945e6841afb6734436277e357a: Deleting consensus metadata
I20250622 13:54:24.210299 7110 catalog_manager.cc:4928] TS cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): tablet 47688cd9cb0a457383d5a27e813b5c0f (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.231526 7619 tablet_service.cc:1515] Processing DeleteTablet for tablet cfff2afb0bc14d3a83aa1bedd156d26d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:57030
I20250622 13:54:24.231859 7662 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:24.232023 7662 raft_consensus.cc:2241] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.232141 7662 pending_rounds.cc:70] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Trying to abort 1 pending ops.
I20250622 13:54:24.232172 7662 pending_rounds.cc:77] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170459496063889408 op_type: NO_OP noop_request { }
I20250622 13:54:24.232254 7662 raft_consensus.cc:2887] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250622 13:54:24.232318 7662 raft_consensus.cc:2270] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.233047 7662 ts_tablet_manager.cc:1905] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.237320 7662 ts_tablet_manager.cc:1918] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250622 13:54:24.237460 7662 log.cc:1199] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root/wals/cfff2afb0bc14d3a83aa1bedd156d26d
I20250622 13:54:24.237752 7662 ts_tablet_manager.cc:1939] T cfff2afb0bc14d3a83aa1bedd156d26d P cc8abf945e6841afb6734436277e357a: Deleting consensus metadata
I20250622 13:54:24.238714 7110 catalog_manager.cc:4928] TS cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): tablet cfff2afb0bc14d3a83aa1bedd156d26d (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.250756 7619 tablet_service.cc:1515] Processing DeleteTablet for tablet 531faa077d23417798976a089cc8c882 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 13:54:24 UTC) from {username='slave'} at 127.0.0.1:57030
I20250622 13:54:24.251058 7662 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:24.251263 7662 raft_consensus.cc:2241] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:24.251452 7662 raft_consensus.cc:2270] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:24.252259 7662 ts_tablet_manager.cc:1905] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 13:54:24.258397 7662 ts_tablet_manager.cc:1918] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250622 13:54:24.258507 7662 log.cc:1199] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root/wals/531faa077d23417798976a089cc8c882
I20250622 13:54:24.258873 7662 ts_tablet_manager.cc:1939] T 531faa077d23417798976a089cc8c882 P cc8abf945e6841afb6734436277e357a: Deleting consensus metadata
I20250622 13:54:24.259788 7110 catalog_manager.cc:4928] TS cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): tablet 531faa077d23417798976a089cc8c882 (table dugtrio [id=d3dd8aa8e838479d880cb8ae91c83344]) successfully deleted
I20250622 13:54:24.326238 7427 raft_consensus.cc:491] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cc8abf945e6841afb6734436277e357a)
I20250622 13:54:24.326447 7427 raft_consensus.cc:513] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.327011 7480 raft_consensus.cc:491] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cc8abf945e6841afb6734436277e357a)
I20250622 13:54:24.327200 7427 leader_election.cc:290] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387)
I20250622 13:54:24.327167 7480 raft_consensus.cc:513] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.327697 7630 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a" is_pre_election: true
I20250622 13:54:24.327708 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071" is_pre_election: true
I20250622 13:54:24.327855 7480 leader_election.cc:290] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:24.327880 7300 raft_consensus.cc:2466] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 1.
I20250622 13:54:24.327884 7630 raft_consensus.cc:2466] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 1.
I20250622 13:54:24.328315 7338 leader_election.cc:304] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071; no voters:
I20250622 13:54:24.328786 7427 raft_consensus.cc:2802] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:54:24.328846 7374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "02e854e07d384b73926562002ae7320f" is_pre_election: true
I20250622 13:54:24.328886 7427 raft_consensus.cc:491] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Starting leader election (detected failure of leader cc8abf945e6841afb6734436277e357a)
I20250622 13:54:24.329015 7427 raft_consensus.cc:3058] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:24.331502 7427 raft_consensus.cc:513] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.331745 7439 consensus_queue.cc:579] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Leader has been unable to successfully communicate with peer cc8abf945e6841afb6734436277e357a for more than 1 seconds (1.415s)
I20250622 13:54:24.332427 7300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f475d5335b4087a1319af1128b2071"
I20250622 13:54:24.332604 7300 raft_consensus.cc:3058] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:24.332983 7630 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "02e854e07d384b73926562002ae7320f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a"
I20250622 13:54:24.333052 7374 raft_consensus.cc:2391] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f475d5335b4087a1319af1128b2071 in current term 2: Already voted for candidate 02e854e07d384b73926562002ae7320f in this term.
I20250622 13:54:24.333109 7630 raft_consensus.cc:3058] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:54:24.334760 7427 leader_election.cc:290] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 2 election: Requested vote from peers cc8abf945e6841afb6734436277e357a (127.0.103.65:41121), c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387)
I20250622 13:54:24.335001 7629 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574aacbfc59044629a25bc7c211dc570" candidate_uuid: "c0f475d5335b4087a1319af1128b2071" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cc8abf945e6841afb6734436277e357a" is_pre_election: true
I20250622 13:54:24.335441 7300 raft_consensus.cc:2466] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 2.
I20250622 13:54:24.335445 7266 leader_election.cc:304] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f475d5335b4087a1319af1128b2071; no voters: 02e854e07d384b73926562002ae7320f, cc8abf945e6841afb6734436277e357a
I20250622 13:54:24.335790 7630 raft_consensus.cc:2466] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e854e07d384b73926562002ae7320f in term 2.
I20250622 13:54:24.335817 7338 leader_election.cc:304] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e854e07d384b73926562002ae7320f, c0f475d5335b4087a1319af1128b2071; no voters:
I20250622 13:54:24.336046 7480 raft_consensus.cc:2747] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250622 13:54:24.336251 7484 raft_consensus.cc:2802] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:54:24.336398 7484 raft_consensus.cc:695] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 LEADER]: Becoming Leader. State: Replica: 02e854e07d384b73926562002ae7320f, State: Running, Role: LEADER
I20250622 13:54:24.336696 7484 consensus_queue.cc:237] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.338164 7374 consensus_queue.cc:237] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:24.339077 7121 catalog_manager.cc:5582] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f reported cstate change: term changed from 1 to 2, leader changed from cc8abf945e6841afb6734436277e357a (127.0.103.65) to 02e854e07d384b73926562002ae7320f (127.0.103.67). New cstate: current_term: 2 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: HEALTHY } } }
I20250622 13:54:24.340487 7630 raft_consensus.cc:1273] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:54:24.341006 7300 raft_consensus.cc:1273] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:54:24.341665 7484 consensus_queue.cc:1035] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:24.342211 7439 consensus_queue.cc:1035] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:24.345077 7484 raft_consensus.cc:2953] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.350317 7109 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24006fadf8ac4ea1bae1a772207cd122 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 13:54:24.350041 7299 raft_consensus.cc:2953] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
W20250622 13:54:24.351466 7338 consensus_peers.cc:487] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f -> Peer 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): Couldn't send request to peer 3f0bb5ab0b90452ba99796f322156573. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24006fadf8ac4ea1bae1a772207cd122. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:24.351986 7121 catalog_manager.cc:5582] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f reported cstate change: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:24.352936 7630 raft_consensus.cc:2953] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.356348 7374 consensus_queue.cc:237] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } }
I20250622 13:54:24.357988 7629 raft_consensus.cc:1273] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:54:24.358507 7484 consensus_queue.cc:1035] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:24.358683 7299 raft_consensus.cc:1273] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:54:24.359212 7484 consensus_queue.cc:1035] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:24.360630 7439 raft_consensus.cc:2953] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } }
I20250622 13:54:24.361395 7629 raft_consensus.cc:2953] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } }
I20250622 13:54:24.364468 7109 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 24006fadf8ac4ea1bae1a772207cd122 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:24.365763 7299 raft_consensus.cc:2953] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } }
I20250622 13:54:24.366106 7121 catalog_manager.cc:5582] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a reported cstate change: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } }
I20250622 13:54:24.374769 7540 tablet_service.cc:1515] Processing DeleteTablet for tablet 24006fadf8ac4ea1bae1a772207cd122 with delete_type TABLET_DATA_TOMBSTONED (TS 3f0bb5ab0b90452ba99796f322156573 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46782
W20250622 13:54:24.375561 7108 catalog_manager.cc:4895] TS 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): delete failed for tablet 24006fadf8ac4ea1bae1a772207cd122 because the tablet was not found. No further retry: Not found: Tablet not found: 24006fadf8ac4ea1bae1a772207cd122
I20250622 13:54:24.385741 7374 consensus_queue.cc:237] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:24.387919 7629 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250622 13:54:24.387986 7338 consensus_peers.cc:487] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f -> Peer 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): Couldn't send request to peer 3f0bb5ab0b90452ba99796f322156573. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f836cb47622a42ba93df50089655d147. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:24.388090 7299 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:54:24.388545 7439 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:24.388833 7484 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:24.391356 7427 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.392153 7299 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.392153 7629 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.394631 7109 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f836cb47622a42ba93df50089655d147 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250622 13:54:24.395928 7122 catalog_manager.cc:5582] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f reported cstate change: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:24.400345 7374 consensus_queue.cc:237] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.401906 7299 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:54:24.401955 7629 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:54:24.402392 7427 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:24.403250 7427 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:24.404484 7439 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } }
I20250622 13:54:24.405120 7299 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } }
I20250622 13:54:24.405851 7629 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } }
I20250622 13:54:24.408803 7109 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f836cb47622a42ba93df50089655d147 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:24.409914 7123 catalog_manager.cc:5582] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a reported cstate change: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } }
I20250622 13:54:24.413183 7540 tablet_service.cc:1515] Processing DeleteTablet for tablet f836cb47622a42ba93df50089655d147 with delete_type TABLET_DATA_TOMBSTONED (TS 3f0bb5ab0b90452ba99796f322156573 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46782
W20250622 13:54:24.413843 7108 catalog_manager.cc:4895] TS 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): delete failed for tablet f836cb47622a42ba93df50089655d147 because the tablet was not found. No further retry: Not found: Tablet not found: f836cb47622a42ba93df50089655d147
I20250622 13:54:24.461665 7480 consensus_queue.cc:579] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Leader has been unable to successfully communicate with peer cc8abf945e6841afb6734436277e357a for more than 1 seconds (1.619s)
I20250622 13:54:24.465570 7299 consensus_queue.cc:237] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:24.468003 7374 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:54:24.468180 7629 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 13:54:24.468638 7480 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:24.468932 7673 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250622 13:54:24.474071 7264 consensus_peers.cc:487] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 -> Peer 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): Couldn't send request to peer 3f0bb5ab0b90452ba99796f322156573. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed5c98b314f340ada36c672fe683ef7a. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:24.474581 7480 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.475234 7374 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.475802 7629 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.479192 7108 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ed5c98b314f340ada36c672fe683ef7a with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 13:54:24.479349 7122 catalog_manager.cc:5582] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f reported cstate change: config changed from index -1 to 2, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "c0f475d5335b4087a1319af1128b2071" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.483980 7299 consensus_queue.cc:237] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } }
I20250622 13:54:24.485531 7629 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:54:24.485558 7374 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250622 13:54:24.486081 7480 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:24.486295 7673 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:24.488255 7678 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } }
I20250622 13:54:24.488976 7374 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } }
I20250622 13:54:24.488945 7629 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } }
I20250622 13:54:24.493429 7123 catalog_manager.cc:5582] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f reported cstate change: config changed from index 2 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "c0f475d5335b4087a1319af1128b2071" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } }
I20250622 13:54:24.493724 7108 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ed5c98b314f340ada36c672fe683ef7a with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:24.496584 7540 tablet_service.cc:1515] Processing DeleteTablet for tablet ed5c98b314f340ada36c672fe683ef7a with delete_type TABLET_DATA_TOMBSTONED (TS 3f0bb5ab0b90452ba99796f322156573 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46782
W20250622 13:54:24.497239 7108 catalog_manager.cc:4895] TS 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): delete failed for tablet ed5c98b314f340ada36c672fe683ef7a because the tablet was not found. No further retry: Not found: Tablet not found: ed5c98b314f340ada36c672fe683ef7a
I20250622 13:54:24.808403 7629 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:54:24.808992 7484 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:24.812635 7299 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:54:24.813069 7427 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:54:24.880195 7299 consensus_queue.cc:237] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:24.882174 7374 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250622 13:54:24.882366 7264 consensus_peers.cc:487] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 -> Peer 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): Couldn't send request to peer 3f0bb5ab0b90452ba99796f322156573. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed5c98b314f340ada36c672fe683ef7a. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:24.882198 7629 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250622 13:54:24.882767 7673 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:24.883018 7480 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:24.885077 7480 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.886209 7374 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.889413 7122 catalog_manager.cc:5582] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 reported cstate change: config changed from index 3 to 4, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "c0f475d5335b4087a1319af1128b2071" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:24.896152 7374 consensus_queue.cc:237] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:24.898120 7299 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250622 13:54:24.898239 7630 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250622 13:54:24.898240 7338 consensus_peers.cc:487] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f -> Peer 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): Couldn't send request to peer 3f0bb5ab0b90452ba99796f322156573. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 574aacbfc59044629a25bc7c211dc570. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:24.898597 7427 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:24.898846 7484 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:54:24.900765 7427 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.901564 7299 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.901602 7630 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.885695 7629 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.905969 7122 catalog_manager.cc:5582] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 reported cstate change: config changed from index -1 to 3, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.907485 7374 consensus_queue.cc:237] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:24.909550 7629 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250622 13:54:24.910102 7338 consensus_peers.cc:487] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f -> Peer 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68:33189): Couldn't send request to peer 3f0bb5ab0b90452ba99796f322156573. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f836cb47622a42ba93df50089655d147. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:54:24.910262 7299 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250622 13:54:24.910372 7484 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:24.911062 7484 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:24.912674 7439 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.913245 7629 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.913677 7299 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } } }
I20250622 13:54:24.917706 7121 catalog_manager.cc:5582] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f reported cstate change: config changed from index 3 to 4, NON_VOTER 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:25.116158 7575 heartbeater.cc:499] Master 127.0.103.126:44101 was elected leader, sending a full tablet report...
I20250622 13:54:25.321863 7687 ts_tablet_manager.cc:927] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: Initiating tablet copy from peer c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387)
I20250622 13:54:25.324476 7687 tablet_copy_client.cc:323] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:45387
I20250622 13:54:25.331075 7310 tablet_copy_service.cc:140] P c0f475d5335b4087a1319af1128b2071: Received BeginTabletCopySession request for tablet ed5c98b314f340ada36c672fe683ef7a from peer 3f0bb5ab0b90452ba99796f322156573 ({username='slave'} at 127.0.0.1:48268)
I20250622 13:54:25.331287 7310 tablet_copy_service.cc:161] P c0f475d5335b4087a1319af1128b2071: Beginning new tablet copy session on tablet ed5c98b314f340ada36c672fe683ef7a from peer 3f0bb5ab0b90452ba99796f322156573 at {username='slave'} at 127.0.0.1:48268: session id = 3f0bb5ab0b90452ba99796f322156573-ed5c98b314f340ada36c672fe683ef7a
I20250622 13:54:25.333177 7690 ts_tablet_manager.cc:927] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: Initiating tablet copy from peer 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:25.333741 7690 tablet_copy_client.cc:323] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33049
I20250622 13:54:25.334709 7310 tablet_copy_source_session.cc:215] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:25.336020 7687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5c98b314f340ada36c672fe683ef7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:25.340287 7384 tablet_copy_service.cc:140] P 02e854e07d384b73926562002ae7320f: Received BeginTabletCopySession request for tablet 574aacbfc59044629a25bc7c211dc570 from peer 3f0bb5ab0b90452ba99796f322156573 ({username='slave'} at 127.0.0.1:49898)
I20250622 13:54:25.340428 7384 tablet_copy_service.cc:161] P 02e854e07d384b73926562002ae7320f: Beginning new tablet copy session on tablet 574aacbfc59044629a25bc7c211dc570 from peer 3f0bb5ab0b90452ba99796f322156573 at {username='slave'} at 127.0.0.1:49898: session id = 3f0bb5ab0b90452ba99796f322156573-574aacbfc59044629a25bc7c211dc570
I20250622 13:54:25.342510 7687 tablet_copy_client.cc:806] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:25.342738 7687 tablet_copy_client.cc:670] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:25.343163 7384 tablet_copy_source_session.cc:215] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:25.344312 7690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574aacbfc59044629a25bc7c211dc570. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:25.345260 7687 tablet_copy_client.cc:538] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:25.347941 7687 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: Bootstrap starting.
I20250622 13:54:25.355402 7690 tablet_copy_client.cc:806] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:25.355607 7690 tablet_copy_client.cc:670] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:25.358822 7687 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:25.359161 7687 tablet_bootstrap.cc:492] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: Bootstrap complete.
I20250622 13:54:25.359382 7687 ts_tablet_manager.cc:1397] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20250622 13:54:25.360059 7687 raft_consensus.cc:357] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:25.360258 7687 raft_consensus.cc:738] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f0bb5ab0b90452ba99796f322156573, State: Initialized, Role: LEARNER
I20250622 13:54:25.360445 7687 consensus_queue.cc:260] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:25.360816 7690 tablet_copy_client.cc:538] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:25.364161 7690 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: Bootstrap starting.
I20250622 13:54:25.366844 7687 ts_tablet_manager.cc:1428] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250622 13:54:25.369896 7310 tablet_copy_service.cc:342] P c0f475d5335b4087a1319af1128b2071: Request end of tablet copy session 3f0bb5ab0b90452ba99796f322156573-ed5c98b314f340ada36c672fe683ef7a received from {username='slave'} at 127.0.0.1:48268
I20250622 13:54:25.370059 7310 tablet_copy_service.cc:434] P c0f475d5335b4087a1319af1128b2071: ending tablet copy session 3f0bb5ab0b90452ba99796f322156573-ed5c98b314f340ada36c672fe683ef7a on tablet ed5c98b314f340ada36c672fe683ef7a with peer 3f0bb5ab0b90452ba99796f322156573
I20250622 13:54:25.371358 7695 ts_tablet_manager.cc:927] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: Initiating tablet copy from peer 02e854e07d384b73926562002ae7320f (127.0.103.67:33049)
I20250622 13:54:25.373587 7695 tablet_copy_client.cc:323] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33049
I20250622 13:54:25.374174 7384 tablet_copy_service.cc:140] P 02e854e07d384b73926562002ae7320f: Received BeginTabletCopySession request for tablet f836cb47622a42ba93df50089655d147 from peer 3f0bb5ab0b90452ba99796f322156573 ({username='slave'} at 127.0.0.1:49898)
I20250622 13:54:25.374279 7384 tablet_copy_service.cc:161] P 02e854e07d384b73926562002ae7320f: Beginning new tablet copy session on tablet f836cb47622a42ba93df50089655d147 from peer 3f0bb5ab0b90452ba99796f322156573 at {username='slave'} at 127.0.0.1:49898: session id = 3f0bb5ab0b90452ba99796f322156573-f836cb47622a42ba93df50089655d147
I20250622 13:54:25.375205 7690 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:25.375571 7690 tablet_bootstrap.cc:492] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: Bootstrap complete.
I20250622 13:54:25.375829 7690 ts_tablet_manager.cc:1397] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20250622 13:54:25.376416 7690 raft_consensus.cc:357] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:25.376600 7690 raft_consensus.cc:738] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3f0bb5ab0b90452ba99796f322156573, State: Initialized, Role: LEARNER
I20250622 13:54:25.376781 7690 consensus_queue.cc:260] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:25.377240 7384 tablet_copy_source_session.cc:215] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:54:25.377382 7690 ts_tablet_manager.cc:1428] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250622 13:54:25.378218 7384 tablet_copy_service.cc:342] P 02e854e07d384b73926562002ae7320f: Request end of tablet copy session 3f0bb5ab0b90452ba99796f322156573-574aacbfc59044629a25bc7c211dc570 received from {username='slave'} at 127.0.0.1:49898
I20250622 13:54:25.378324 7384 tablet_copy_service.cc:434] P 02e854e07d384b73926562002ae7320f: ending tablet copy session 3f0bb5ab0b90452ba99796f322156573-574aacbfc59044629a25bc7c211dc570 on tablet 574aacbfc59044629a25bc7c211dc570 with peer 3f0bb5ab0b90452ba99796f322156573
I20250622 13:54:25.378669 7695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f836cb47622a42ba93df50089655d147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:54:25.384873 7695 tablet_copy_client.cc:806] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Starting download of 0 data blocks...
I20250622 13:54:25.385190 7695 tablet_copy_client.cc:670] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Starting download of 1 WAL segments...
I20250622 13:54:25.386619 7695 tablet_copy_client.cc:538] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:54:25.389938 7695 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: Bootstrap starting.
I20250622 13:54:25.396749 7695 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:54:25.397146 7695 tablet_bootstrap.cc:492] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: Bootstrap complete.
I20250622 13:54:25.397426 7695 ts_tablet_manager.cc:1397] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250622 13:54:25.398131 7695 raft_consensus.cc:357] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:25.398341 7695 raft_consensus.cc:738] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f0bb5ab0b90452ba99796f322156573, State: Initialized, Role: LEARNER
I20250622 13:54:25.398519 7695 consensus_queue.cc:260] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: true } }
I20250622 13:54:25.398926 7695 ts_tablet_manager.cc:1428] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250622 13:54:25.399874 7384 tablet_copy_service.cc:342] P 02e854e07d384b73926562002ae7320f: Request end of tablet copy session 3f0bb5ab0b90452ba99796f322156573-f836cb47622a42ba93df50089655d147 received from {username='slave'} at 127.0.0.1:49898
I20250622 13:54:25.399956 7384 tablet_copy_service.cc:434] P 02e854e07d384b73926562002ae7320f: ending tablet copy session 3f0bb5ab0b90452ba99796f322156573-f836cb47622a42ba93df50089655d147 on tablet f836cb47622a42ba93df50089655d147 with peer 3f0bb5ab0b90452ba99796f322156573
I20250622 13:54:25.718482 7550 raft_consensus.cc:1215] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250622 13:54:25.904596 7550 raft_consensus.cc:1215] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250622 13:54:25.962311 7550 raft_consensus.cc:1215] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250622 13:54:26.189517 7699 raft_consensus.cc:1062] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071: attempting to promote NON_VOTER 3f0bb5ab0b90452ba99796f322156573 to VOTER
I20250622 13:54:26.190390 7699 consensus_queue.cc:237] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } }
I20250622 13:54:26.192632 7629 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250622 13:54:26.192628 7373 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250622 13:54:26.192628 7550 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 LEARNER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250622 13:54:26.193219 7698 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:26.193466 7673 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:26.193773 7699 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:26.198549 7673 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.199299 7373 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.202430 7629 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.207291 7122 catalog_manager.cc:5582] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 reported cstate change: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f475d5335b4087a1319af1128b2071" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:54:26.210484 7550 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } attrs { replace: true } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.220203 7300 consensus_queue.cc:237] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } }
I20250622 13:54:26.221673 7550 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250622 13:54:26.222143 7629 raft_consensus.cc:1273] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer c0f475d5335b4087a1319af1128b2071: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250622 13:54:26.222096 7699 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:54:26.222740 7699 consensus_queue.cc:1035] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:54:26.224509 7673 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02e854e07d384b73926562002ae7320f (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.225288 7629 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02e854e07d384b73926562002ae7320f (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.225383 7550 raft_consensus.cc:2953] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02e854e07d384b73926562002ae7320f (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.228158 7108 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ed5c98b314f340ada36c672fe683ef7a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:26.229218 7123 catalog_manager.cc:5582] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 reported cstate change: config changed from index 5 to 6, VOTER 02e854e07d384b73926562002ae7320f (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "c0f475d5335b4087a1319af1128b2071" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:54:26.233240 7363 tablet_service.cc:1515] Processing DeleteTablet for tablet ed5c98b314f340ada36c672fe683ef7a with delete_type TABLET_DATA_TOMBSTONED (TS 02e854e07d384b73926562002ae7320f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38692
I20250622 13:54:26.233927 7712 tablet_replica.cc:331] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: stopping tablet replica
I20250622 13:54:26.234201 7712 raft_consensus.cc:2241] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:26.234493 7712 raft_consensus.cc:2270] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:26.235332 7712 ts_tablet_manager.cc:1905] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:54:26.240909 7712 ts_tablet_manager.cc:1918] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250622 13:54:26.241035 7712 log.cc:1199] T ed5c98b314f340ada36c672fe683ef7a P 02e854e07d384b73926562002ae7320f: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-2-root/wals/ed5c98b314f340ada36c672fe683ef7a
I20250622 13:54:26.241633 7109 catalog_manager.cc:4928] TS 02e854e07d384b73926562002ae7320f (127.0.103.67:33049): tablet ed5c98b314f340ada36c672fe683ef7a (table test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]) successfully deleted
I20250622 13:54:26.306433 7427 raft_consensus.cc:1062] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f: attempting to promote NON_VOTER 3f0bb5ab0b90452ba99796f322156573 to VOTER
I20250622 13:54:26.307189 7427 consensus_queue.cc:237] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } }
I20250622 13:54:26.309299 7550 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 LEARNER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:26.309299 7299 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:26.309592 7629 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250622 13:54:26.309983 7700 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:26.310490 7700 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:26.310918 7700 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250622 13:54:26.313841 7701 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.317559 7629 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.317623 7550 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.320694 7299 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.321753 7121 catalog_manager.cc:5582] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f reported cstate change: config changed from index 3 to 4, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:54:26.326324 7374 consensus_queue.cc:237] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } }
I20250622 13:54:26.328635 7550 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250622 13:54:26.328635 7629 raft_consensus.cc:1273] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250622 13:54:26.330420 7427 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:26.330906 7427 consensus_queue.cc:1035] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:26.333063 7550 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c0f475d5335b4087a1319af1128b2071 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.333253 7629 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c0f475d5335b4087a1319af1128b2071 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.332497 7427 raft_consensus.cc:2953] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c0f475d5335b4087a1319af1128b2071 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.337065 7120 catalog_manager.cc:5582] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 reported cstate change: config changed from index 4 to 5, VOTER c0f475d5335b4087a1319af1128b2071 (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.339607 7109 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 574aacbfc59044629a25bc7c211dc570 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:26.340219 7290 tablet_service.cc:1515] Processing DeleteTablet for tablet 574aacbfc59044629a25bc7c211dc570 with delete_type TABLET_DATA_TOMBSTONED (TS c0f475d5335b4087a1319af1128b2071 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40126
I20250622 13:54:26.340822 7718 tablet_replica.cc:331] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: stopping tablet replica
I20250622 13:54:26.341053 7718 raft_consensus.cc:2241] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:26.341295 7718 raft_consensus.cc:2270] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:26.342072 7718 ts_tablet_manager.cc:1905] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:54:26.347180 7718 ts_tablet_manager.cc:1918] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250622 13:54:26.347275 7718 log.cc:1199] T 574aacbfc59044629a25bc7c211dc570 P c0f475d5335b4087a1319af1128b2071: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-1-root/wals/574aacbfc59044629a25bc7c211dc570
I20250622 13:54:26.347872 7108 catalog_manager.cc:4928] TS c0f475d5335b4087a1319af1128b2071 (127.0.103.66:45387): tablet 574aacbfc59044629a25bc7c211dc570 (table test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]) successfully deleted
I20250622 13:54:26.536170 7427 raft_consensus.cc:1062] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f: attempting to promote NON_VOTER 3f0bb5ab0b90452ba99796f322156573 to VOTER
I20250622 13:54:26.536866 7427 consensus_queue.cc:237] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } }
I20250622 13:54:26.538822 7629 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250622 13:54:26.538908 7550 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 LEARNER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250622 13:54:26.538822 7299 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250622 13:54:26.539332 7427 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:26.539551 7700 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:26.539809 7714 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:54:26.542953 7427 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.543501 7550 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.543684 7299 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.547385 7629 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.547917 7121 catalog_manager.cc:5582] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 reported cstate change: config changed from index 4 to 5, 3f0bb5ab0b90452ba99796f322156573 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cc8abf945e6841afb6734436277e357a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41121 } attrs { replace: true } } peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
W20250622 13:54:26.553236 7427 consensus_queue.cc:1162] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cc8abf945e6841afb6734436277e357a" responder_term: 1 status { last_received { term: 1 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 5 } }
I20250622 13:54:26.553586 7373 consensus_queue.cc:237] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } }
I20250622 13:54:26.554894 7550 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250622 13:54:26.555274 7299 raft_consensus.cc:1273] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Refusing update from remote peer 02e854e07d384b73926562002ae7320f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250622 13:54:26.555390 7701 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:54:26.556397 7701 consensus_queue.cc:1035] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:54:26.557987 7427 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc8abf945e6841afb6734436277e357a (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.558761 7299 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc8abf945e6841afb6734436277e357a (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.558759 7549 raft_consensus.cc:2953] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cc8abf945e6841afb6734436277e357a (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.562269 7109 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f836cb47622a42ba93df50089655d147 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:54:26.562847 7123 catalog_manager.cc:5582] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 reported cstate change: config changed from index 5 to 6, VOTER cc8abf945e6841afb6734436277e357a (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "02e854e07d384b73926562002ae7320f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0f475d5335b4087a1319af1128b2071" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 45387 } } peers { permanent_uuid: "02e854e07d384b73926562002ae7320f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33049 } } peers { permanent_uuid: "3f0bb5ab0b90452ba99796f322156573" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33189 } attrs { promote: false } } }
I20250622 13:54:26.566201 7619 tablet_service.cc:1515] Processing DeleteTablet for tablet f836cb47622a42ba93df50089655d147 with delete_type TABLET_DATA_TOMBSTONED (TS cc8abf945e6841afb6734436277e357a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57030
I20250622 13:54:26.566954 7723 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:26.567113 7723 raft_consensus.cc:2241] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:26.567370 7723 raft_consensus.cc:2270] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:26.568209 7723 ts_tablet_manager.cc:1905] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:54:26.572677 7723 ts_tablet_manager.cc:1918] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250622 13:54:26.572765 7723 log.cc:1199] T f836cb47622a42ba93df50089655d147 P cc8abf945e6841afb6734436277e357a: Deleting WAL directory at /tmp/dist-test-taskgowkLS/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750600281110842-413-0/minicluster-data/ts-0-root/wals/f836cb47622a42ba93df50089655d147
I20250622 13:54:26.573426 7110 catalog_manager.cc:4928] TS cc8abf945e6841afb6734436277e357a (127.0.103.65:41121): tablet f836cb47622a42ba93df50089655d147 (table test-workload [id=6d5fdf99bb7f4be7acfbaae9ec4f1084]) successfully deleted
W20250622 13:54:27.563472 7171 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver c0f475d5335b4087a1319af1128b2071 not available for placement
I20250622 13:54:28.459837 413 tablet_server.cc:178] TabletServer@127.0.103.65:41121 shutting down...
I20250622 13:54:28.480664 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:28.480963 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:28.481122 413 raft_consensus.cc:2241] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:28.481468 413 raft_consensus.cc:2270] T ed5c98b314f340ada36c672fe683ef7a P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.482147 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:28.482280 413 raft_consensus.cc:2241] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:28.482484 413 raft_consensus.cc:2270] T 574aacbfc59044629a25bc7c211dc570 P cc8abf945e6841afb6734436277e357a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.483556 413 tablet_replica.cc:331] stopping tablet replica
I20250622 13:54:28.483682 413 raft_consensus.cc:2241] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:28.483852 413 raft_consensus.cc:2270] T 24006fadf8ac4ea1bae1a772207cd122 P cc8abf945e6841afb6734436277e357a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.497859 413 tablet_server.cc:195] TabletServer@127.0.103.65:41121 shutdown complete.
I20250622 13:54:28.501919 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250622 13:54:28.510591 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:28.510857 413 tablet_replica.cc:331] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071: stopping tablet replica
I20250622 13:54:28.511029 413 raft_consensus.cc:2241] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:28.511332 413 raft_consensus.cc:2270] T f836cb47622a42ba93df50089655d147 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.511912 413 tablet_replica.cc:331] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071: stopping tablet replica
I20250622 13:54:28.512044 413 raft_consensus.cc:2241] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:28.512303 413 raft_consensus.cc:2270] T ed5c98b314f340ada36c672fe683ef7a P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.512740 413 tablet_replica.cc:331] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071: stopping tablet replica
I20250622 13:54:28.512843 413 raft_consensus.cc:2241] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:28.513031 413 raft_consensus.cc:2270] T 24006fadf8ac4ea1bae1a772207cd122 P c0f475d5335b4087a1319af1128b2071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.527153 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250622 13:54:28.531327 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250622 13:54:28.539901 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:28.540141 413 tablet_replica.cc:331] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f: stopping tablet replica
I20250622 13:54:28.540349 413 raft_consensus.cc:2241] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:28.540640 413 raft_consensus.cc:2270] T 24006fadf8ac4ea1bae1a772207cd122 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.541185 413 tablet_replica.cc:331] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f: stopping tablet replica
I20250622 13:54:28.541289 413 raft_consensus.cc:2241] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:28.541540 413 raft_consensus.cc:2270] T f836cb47622a42ba93df50089655d147 P 02e854e07d384b73926562002ae7320f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.541970 413 tablet_replica.cc:331] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f: stopping tablet replica
I20250622 13:54:28.542063 413 raft_consensus.cc:2241] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 LEADER]: Raft consensus shutting down.
I20250622 13:54:28.542284 413 raft_consensus.cc:2270] T 574aacbfc59044629a25bc7c211dc570 P 02e854e07d384b73926562002ae7320f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.556263 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
W20250622 13:54:28.564244 7171 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver cc8abf945e6841afb6734436277e357a not available for placement
I20250622 13:54:28.569054 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250622 13:54:28.577831 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250622 13:54:28.578087 413 tablet_replica.cc:331] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573: stopping tablet replica
I20250622 13:54:28.578269 413 raft_consensus.cc:2241] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:28.578516 413 raft_consensus.cc:2270] T f836cb47622a42ba93df50089655d147 P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.579110 413 tablet_replica.cc:331] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573: stopping tablet replica
I20250622 13:54:28.579221 413 raft_consensus.cc:2241] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:28.579373 413 raft_consensus.cc:2270] T ed5c98b314f340ada36c672fe683ef7a P 3f0bb5ab0b90452ba99796f322156573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.579771 413 tablet_replica.cc:331] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573: stopping tablet replica
I20250622 13:54:28.579862 413 raft_consensus.cc:2241] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:54:28.580011 413 raft_consensus.cc:2270] T 574aacbfc59044629a25bc7c211dc570 P 3f0bb5ab0b90452ba99796f322156573 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.593750 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250622 13:54:28.597033 413 master.cc:561] Master@127.0.103.126:44101 shutting down...
I20250622 13:54:28.604331 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 1 LEADER]: Raft consensus shutting down.
I20250622 13:54:28.604557 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 13:54:28.604653 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P cf3d5833c39d490d82e5b74e1dba281c: stopping tablet replica
I20250622 13:54:28.618095 413 master.cc:583] Master@127.0.103.126:44101 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (7818 ms)
[----------] 14 tests from AutoRebalancerTest (187452 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (187452 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20250622 13:54:28.631671 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20250622 13:54:28.631733 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20250622 13:54:28.631764 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:4671: suppressed but not reported on 11 messages since previous log ~4 seconds ago
I20250622 13:54:28.631814 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:5260: suppressed but not reported on 17 messages since previous log ~30 seconds ago
I20250622 13:54:28.631860 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 8 messages since previous log ~173 seconds ago
I20250622 13:54:28.631943 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 49 messages since previous log ~17 seconds ago
I20250622 13:54:28.631991 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 16 messages since previous log ~4 seconds ago